2023-04-25 02:53:34,278 DEBUG [main] hbase.HBaseTestingUtility(342): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/68daf6bb-f478-d813-e37b-dbe23bee4a08 2023-04-25 02:53:34,291 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.TestAcidGuaranteesWithEagerPolicy timeout: 13 mins 2023-04-25 02:53:34,305 INFO [Time-limited test] hbase.HBaseTestingUtility(1068): Starting up minicluster with option: StartMiniClusterOption{numMasters=1, masterClass=null, numRegionServers=1, rsPorts=, rsClass=null, numDataNodes=1, dataNodeHosts=null, numZkServers=1, createRootDir=false, createWALDir=false} 2023-04-25 02:53:34,306 INFO [Time-limited test] hbase.HBaseZKTestingUtility(82): Created new mini-cluster data directory: /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/68daf6bb-f478-d813-e37b-dbe23bee4a08/cluster_35ab5f43-d241-074f-505d-0f2e954a1696, deleteOnExit=true 2023-04-25 02:53:34,306 INFO [Time-limited test] hbase.HBaseTestingUtility(1082): STARTING DFS 2023-04-25 02:53:34,307 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting test.cache.data to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/68daf6bb-f478-d813-e37b-dbe23bee4a08/test.cache.data in system properties and HBase conf 2023-04-25 02:53:34,307 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.tmp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/68daf6bb-f478-d813-e37b-dbe23bee4a08/hadoop.tmp.dir in system properties and HBase conf 2023-04-25 02:53:34,308 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.log.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/68daf6bb-f478-d813-e37b-dbe23bee4a08/hadoop.log.dir in system properties and HBase conf 2023-04-25 02:53:34,308 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.local.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/68daf6bb-f478-d813-e37b-dbe23bee4a08/mapreduce.cluster.local.dir in system properties and HBase conf 2023-04-25 02:53:34,308 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.temp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/68daf6bb-f478-d813-e37b-dbe23bee4a08/mapreduce.cluster.temp.dir in system properties and HBase conf 2023-04-25 02:53:34,308 INFO [Time-limited test] hbase.HBaseTestingUtility(759): read short circuit is OFF 2023-04-25 02:53:34,440 WARN [Time-limited test] util.NativeCodeLoader(62): Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2023-04-25 02:53:34,857 DEBUG [Time-limited test] fs.HFileSystem(308): The file system is not a DistributedFileSystem. Skipping on block location reordering 2023-04-25 02:53:34,861 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-labels.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/68daf6bb-f478-d813-e37b-dbe23bee4a08/yarn.node-labels.fs-store.root-dir in system properties and HBase conf 2023-04-25 02:53:34,862 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-attribute.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/68daf6bb-f478-d813-e37b-dbe23bee4a08/yarn.node-attribute.fs-store.root-dir in system properties and HBase conf 2023-04-25 02:53:34,862 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.log-dirs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/68daf6bb-f478-d813-e37b-dbe23bee4a08/yarn.nodemanager.log-dirs in system properties and HBase conf 2023-04-25 02:53:34,862 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/68daf6bb-f478-d813-e37b-dbe23bee4a08/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-04-25 02:53:34,862 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.active-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/68daf6bb-f478-d813-e37b-dbe23bee4a08/yarn.timeline-service.entity-group-fs-store.active-dir in system properties and HBase conf 2023-04-25 02:53:34,863 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.done-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/68daf6bb-f478-d813-e37b-dbe23bee4a08/yarn.timeline-service.entity-group-fs-store.done-dir in system properties and HBase conf 2023-04-25 02:53:34,863 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/68daf6bb-f478-d813-e37b-dbe23bee4a08/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-04-25 02:53:34,863 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/68daf6bb-f478-d813-e37b-dbe23bee4a08/dfs.journalnode.edits.dir in system properties and HBase conf 2023-04-25 02:53:34,864 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.datanode.shared.file.descriptor.paths to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/68daf6bb-f478-d813-e37b-dbe23bee4a08/dfs.datanode.shared.file.descriptor.paths in system properties and HBase conf 2023-04-25 02:53:34,864 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting nfs.dump.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/68daf6bb-f478-d813-e37b-dbe23bee4a08/nfs.dump.dir in system properties and HBase conf 2023-04-25 02:53:34,865 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting java.io.tmpdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/68daf6bb-f478-d813-e37b-dbe23bee4a08/java.io.tmpdir in system properties and HBase conf 2023-04-25 02:53:34,865 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/68daf6bb-f478-d813-e37b-dbe23bee4a08/dfs.journalnode.edits.dir in system properties and HBase conf 2023-04-25 02:53:34,865 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.provided.aliasmap.inmemory.leveldb.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/68daf6bb-f478-d813-e37b-dbe23bee4a08/dfs.provided.aliasmap.inmemory.leveldb.dir in system properties and HBase conf 2023-04-25 02:53:34,865 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting fs.s3a.committer.staging.tmp.path to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/68daf6bb-f478-d813-e37b-dbe23bee4a08/fs.s3a.committer.staging.tmp.path in system properties and HBase conf Formatting using clusterid: testClusterID 2023-04-25 02:53:35,428 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-04-25 02:53:35,433 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-04-25 02:53:35,742 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties 2023-04-25 02:53:35,945 INFO [Time-limited test] log.Slf4jLog(67): Logging to org.slf4j.impl.Reload4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog 2023-04-25 02:53:35,962 WARN [Time-limited test] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-04-25 02:53:36,000 INFO [Time-limited test] log.Slf4jLog(67): jetty-6.1.26 2023-04-25 02:53:36,036 INFO [Time-limited test] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/hdfs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/68daf6bb-f478-d813-e37b-dbe23bee4a08/java.io.tmpdir/Jetty_localhost_46331_hdfs____.ehzrb/webapp 2023-04-25 02:53:36,181 INFO [Time-limited test] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:46331 2023-04-25 02:53:36,219 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-04-25 02:53:36,219 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-04-25 02:53:36,694 WARN [Listener at localhost/43711] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-04-25 02:53:36,820 WARN [Listener at localhost/43711] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-04-25 02:53:36,844 WARN [Listener at localhost/43711] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-04-25 02:53:36,852 INFO [Listener at localhost/43711] log.Slf4jLog(67): jetty-6.1.26 2023-04-25 02:53:36,860 INFO [Listener at localhost/43711] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/68daf6bb-f478-d813-e37b-dbe23bee4a08/java.io.tmpdir/Jetty_localhost_36669_datanode____.458xr7/webapp 2023-04-25 02:53:36,986 INFO [Listener at localhost/43711] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:36669 2023-04-25 02:53:37,338 WARN [Listener at localhost/41915] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-04-25 02:53:37,882 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x59c6b8e080f08994: Processing first storage report for DS-db6f9fb0-c5d9-4913-bf97-d523392806fc from datanode 2543a0b5-98fe-4585-af6e-bed4c7220585 2023-04-25 02:53:37,884 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x59c6b8e080f08994: from storage DS-db6f9fb0-c5d9-4913-bf97-d523392806fc node DatanodeRegistration(127.0.0.1:35477, datanodeUuid=2543a0b5-98fe-4585-af6e-bed4c7220585, infoPort=40779, infoSecurePort=0, ipcPort=41915, storageInfo=lv=-57;cid=testClusterID;nsid=1020470658;c=1682391215512), blocks: 0, hasStaleStorage: true, processing time: 2 msecs, invalidatedBlocks: 0 2023-04-25 02:53:37,884 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x59c6b8e080f08994: Processing first storage report for DS-43ab3742-2c76-453c-9ec4-2385baba87ac from datanode 2543a0b5-98fe-4585-af6e-bed4c7220585 2023-04-25 02:53:37,884 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x59c6b8e080f08994: from storage DS-43ab3742-2c76-453c-9ec4-2385baba87ac node DatanodeRegistration(127.0.0.1:35477, datanodeUuid=2543a0b5-98fe-4585-af6e-bed4c7220585, infoPort=40779, infoSecurePort=0, ipcPort=41915, storageInfo=lv=-57;cid=testClusterID;nsid=1020470658;c=1682391215512), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-04-25 02:53:37,936 DEBUG [Listener at localhost/41915] hbase.HBaseTestingUtility(649): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/68daf6bb-f478-d813-e37b-dbe23bee4a08 2023-04-25 02:53:38,006 INFO [Listener at localhost/41915] zookeeper.MiniZooKeeperCluster(258): Started connectionTimeout=30000, dir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/68daf6bb-f478-d813-e37b-dbe23bee4a08/cluster_35ab5f43-d241-074f-505d-0f2e954a1696/zookeeper_0, clientPort=62054, secureClientPort=-1, dataDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/68daf6bb-f478-d813-e37b-dbe23bee4a08/cluster_35ab5f43-d241-074f-505d-0f2e954a1696/zookeeper_0/version-2, dataDirSize=424 dataLogDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/68daf6bb-f478-d813-e37b-dbe23bee4a08/cluster_35ab5f43-d241-074f-505d-0f2e954a1696/zookeeper_0/version-2, dataLogSize=424 tickTime=2000, maxClientCnxns=300, minSessionTimeout=4000, maxSessionTimeout=40000, serverId=0 2023-04-25 02:53:38,019 INFO [Listener at localhost/41915] zookeeper.MiniZooKeeperCluster(283): Started MiniZooKeeperCluster and ran 'stat' on client port=62054 2023-04-25 02:53:38,030 INFO [Listener at localhost/41915] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-25 02:53:38,032 INFO [Listener at localhost/41915] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-25 02:53:38,696 INFO [Listener at localhost/41915] util.FSUtils(471): Created version file at hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f with version=8 2023-04-25 02:53:38,696 INFO [Listener at localhost/41915] hbase.HBaseTestingUtility(1406): Setting hbase.fs.tmp.dir to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/hbase-staging 2023-04-25 02:53:38,995 INFO [Listener at localhost/41915] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-04-25 02:53:39,461 INFO [Listener at localhost/41915] client.ConnectionUtils(127): master/jenkins-hbase4:0 server-side Connection retries=45 2023-04-25 02:53:39,492 INFO [Listener at localhost/41915] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-04-25 02:53:39,493 INFO [Listener at localhost/41915] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-04-25 02:53:39,493 INFO [Listener at localhost/41915] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-04-25 02:53:39,493 INFO [Listener at localhost/41915] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-04-25 02:53:39,494 INFO [Listener at localhost/41915] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-04-25 02:53:39,645 INFO [Listener at localhost/41915] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.HbckService, hbase.pb.ClientMetaService, hbase.pb.ClientService, hbase.pb.AdminService 2023-04-25 02:53:39,727 DEBUG [Listener at localhost/41915] util.ClassSize(228): Using Unsafe to estimate memory layout 2023-04-25 02:53:39,826 INFO [Listener at localhost/41915] ipc.NettyRpcServer(120): Bind to /172.31.14.131:44265 2023-04-25 02:53:39,836 INFO [Listener at localhost/41915] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-25 02:53:39,838 INFO [Listener at localhost/41915] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-25 02:53:39,859 INFO [Listener at localhost/41915] zookeeper.RecoverableZooKeeper(93): Process identifier=master:44265 connecting to ZooKeeper ensemble=127.0.0.1:62054 2023-04-25 02:53:39,898 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:442650x0, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-04-25 02:53:39,901 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): master:44265-0x101b93ca1070000 connected 2023-04-25 02:53:39,934 DEBUG [Listener at localhost/41915] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-04-25 02:53:39,935 DEBUG [Listener at localhost/41915] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-04-25 02:53:39,939 DEBUG [Listener at localhost/41915] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-04-25 02:53:39,948 DEBUG [Listener at localhost/41915] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=44265 2023-04-25 02:53:39,948 DEBUG [Listener at localhost/41915] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=44265 2023-04-25 02:53:39,948 DEBUG [Listener at localhost/41915] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=44265 2023-04-25 02:53:39,950 DEBUG [Listener at localhost/41915] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=44265 2023-04-25 02:53:39,951 DEBUG [Listener at localhost/41915] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=44265 2023-04-25 02:53:39,958 INFO [Listener at localhost/41915] master.HMaster(444): hbase.rootdir=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f, hbase.cluster.distributed=false 2023-04-25 02:53:40,024 INFO [Listener at localhost/41915] client.ConnectionUtils(127): regionserver/jenkins-hbase4:0 server-side Connection retries=45 2023-04-25 02:53:40,024 INFO [Listener at localhost/41915] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-04-25 02:53:40,024 INFO [Listener at localhost/41915] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-04-25 02:53:40,025 INFO [Listener at localhost/41915] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-04-25 02:53:40,025 INFO [Listener at localhost/41915] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-04-25 02:53:40,025 INFO [Listener at localhost/41915] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-04-25 02:53:40,030 INFO [Listener at localhost/41915] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-04-25 02:53:40,032 INFO [Listener at localhost/41915] ipc.NettyRpcServer(120): Bind to /172.31.14.131:46697 2023-04-25 02:53:40,035 INFO [Listener at localhost/41915] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-04-25 02:53:40,041 DEBUG [Listener at localhost/41915] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-04-25 02:53:40,041 INFO [Listener at localhost/41915] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-25 02:53:40,045 INFO [Listener at localhost/41915] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-25 02:53:40,046 INFO [Listener at localhost/41915] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:46697 connecting to ZooKeeper ensemble=127.0.0.1:62054 2023-04-25 02:53:40,050 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:466970x0, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-04-25 02:53:40,051 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): regionserver:46697-0x101b93ca1070001 connected 2023-04-25 02:53:40,051 DEBUG [Listener at localhost/41915] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-04-25 02:53:40,052 DEBUG [Listener at localhost/41915] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-04-25 02:53:40,053 DEBUG [Listener at localhost/41915] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-04-25 02:53:40,054 DEBUG [Listener at localhost/41915] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=46697 2023-04-25 02:53:40,054 DEBUG [Listener at localhost/41915] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=46697 2023-04-25 02:53:40,054 DEBUG [Listener at localhost/41915] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=46697 2023-04-25 02:53:40,055 DEBUG [Listener at localhost/41915] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=46697 2023-04-25 02:53:40,055 DEBUG [Listener at localhost/41915] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=46697 2023-04-25 02:53:40,059 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2168): Adding backup master ZNode /hbase/backup-masters/jenkins-hbase4.apache.org,44265,1682391218835 2023-04-25 02:53:40,072 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-04-25 02:53:40,074 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/jenkins-hbase4.apache.org,44265,1682391218835 2023-04-25 02:53:40,098 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-04-25 02:53:40,099 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-04-25 02:53:40,099 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-25 02:53:40,100 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-04-25 02:53:40,101 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(227): Deleting ZNode for /hbase/backup-masters/jenkins-hbase4.apache.org,44265,1682391218835 from backup master directory 2023-04-25 02:53:40,101 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-04-25 02:53:40,104 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/jenkins-hbase4.apache.org,44265,1682391218835 2023-04-25 02:53:40,105 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-04-25 02:53:40,105 WARN [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-04-25 02:53:40,106 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(237): Registered as active master=jenkins-hbase4.apache.org,44265,1682391218835 2023-04-25 02:53:40,109 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating data MemStoreChunkPool with chunk size 2 MB, max count 352, initial count 0 2023-04-25 02:53:40,110 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating index MemStoreChunkPool with chunk size 204.80 KB, max count 391, initial count 0 2023-04-25 02:53:40,196 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] util.FSUtils(620): Created cluster ID file at hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/hbase.id with ID: 42edb776-2c8f-44f4-8800-1527a17d0d3b 2023-04-25 02:53:40,241 INFO [master/jenkins-hbase4:0:becomeActiveMaster] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-25 02:53:40,258 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-25 02:53:40,303 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ReadOnlyZKClient(139): Connect 0x017faac8 to 127.0.0.1:62054 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-25 02:53:40,337 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5db2f714, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-25 02:53:40,361 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(309): Create or load local region for table 'master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-25 02:53:40,364 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(132): Injected flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000 2023-04-25 02:53:40,384 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(264): ClientProtocol::create wrong number of arguments, should be hadoop 3.2 or below 2023-04-25 02:53:40,384 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(270): ClientProtocol::create wrong number of arguments, should be hadoop 2.x 2023-04-25 02:53:40,386 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(279): can not find SHOULD_REPLICATE flag, should be hadoop 2.x java.lang.IllegalArgumentException: No enum constant org.apache.hadoop.fs.CreateFlag.SHOULD_REPLICATE at java.lang.Enum.valueOf(Enum.java:238) at org.apache.hadoop.fs.CreateFlag.valueOf(CreateFlag.java:63) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.loadShouldReplicateFlag(FanOutOneBlockAsyncDFSOutputHelper.java:277) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.(FanOutOneBlockAsyncDFSOutputHelper.java:304) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:139) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-04-25 02:53:40,391 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(243): No decryptEncryptedDataEncryptionKey method in DFSClient, should be hadoop version with HDFS-12396 java.lang.NoSuchMethodException: org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(org.apache.hadoop.fs.FileEncryptionInfo) at java.lang.Class.getDeclaredMethod(Class.java:2130) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelperWithoutHDFS12396(FanOutOneBlockAsyncDFSOutputSaslHelper.java:182) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelper(FanOutOneBlockAsyncDFSOutputSaslHelper.java:241) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.(FanOutOneBlockAsyncDFSOutputSaslHelper.java:252) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:140) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-04-25 02:53:40,392 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-04-25 02:53:40,427 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7693): Creating {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, under table dir hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/MasterData/data/master/store-tmp 2023-04-25 02:53:40,456 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-25 02:53:40,456 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-04-25 02:53:40,456 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-25 02:53:40,457 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-25 02:53:40,457 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-04-25 02:53:40,457 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-25 02:53:40,457 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-25 02:53:40,457 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-04-25 02:53:40,459 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(191): WALDir=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/MasterData/WALs/jenkins-hbase4.apache.org,44265,1682391218835 2023-04-25 02:53:40,482 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C44265%2C1682391218835, suffix=, logDir=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/MasterData/WALs/jenkins-hbase4.apache.org,44265,1682391218835, archiveDir=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/MasterData/oldWALs, maxLogs=10 2023-04-25 02:53:40,538 DEBUG [RS-EventLoopGroup-3-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:35477,DS-db6f9fb0-c5d9-4913-bf97-d523392806fc,DISK] 2023-04-25 02:53:40,546 DEBUG [RS-EventLoopGroup-3-2] asyncfs.ProtobufDecoder(123): Hadoop 3.2 and below use unshaded protobuf. java.lang.ClassNotFoundException: org.apache.hadoop.thirdparty.protobuf.MessageLite at java.net.URLClassLoader.findClass(URLClassLoader.java:387) at java.lang.ClassLoader.loadClass(ClassLoader.java:418) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) at java.lang.ClassLoader.loadClass(ClassLoader.java:351) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.io.asyncfs.ProtobufDecoder.(ProtobufDecoder.java:118) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.processWriteBlockResponse(FanOutOneBlockAsyncDFSOutputHelper.java:340) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$100(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$4.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:424) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:557) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:185) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.initialize(FanOutOneBlockAsyncDFSOutputHelper.java:418) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$300(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:476) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:471) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:625) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:105) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:84) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.fulfillConnectPromise(AbstractEpollChannel.java:653) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:691) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-25 02:53:40,611 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/MasterData/WALs/jenkins-hbase4.apache.org,44265,1682391218835/jenkins-hbase4.apache.org%2C44265%2C1682391218835.1682391220493 2023-04-25 02:53:40,611 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:35477,DS-db6f9fb0-c5d9-4913-bf97-d523392806fc,DISK]] 2023-04-25 02:53:40,612 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7854): Opening region: {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''} 2023-04-25 02:53:40,613 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-25 02:53:40,616 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7894): checking encryption for 1595e783b53d99cd5eef43b6debb2682 2023-04-25 02:53:40,617 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7897): checking classloading for 1595e783b53d99cd5eef43b6debb2682 2023-04-25 02:53:40,682 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family proc of region 1595e783b53d99cd5eef43b6debb2682 2023-04-25 02:53:40,690 DEBUG [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc 2023-04-25 02:53:40,722 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1595e783b53d99cd5eef43b6debb2682 columnFamilyName proc 2023-04-25 02:53:40,737 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(310): Store=1595e783b53d99cd5eef43b6debb2682/proc, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-25 02:53:40,742 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-04-25 02:53:40,745 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-04-25 02:53:40,763 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1055): writing seq id for 1595e783b53d99cd5eef43b6debb2682 2023-04-25 02:53:40,767 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-25 02:53:40,768 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1072): Opened 1595e783b53d99cd5eef43b6debb2682; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=61982546, jitterRate=-0.07638809084892273}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-04-25 02:53:40,768 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(965): Region open journal for 1595e783b53d99cd5eef43b6debb2682: 2023-04-25 02:53:40,769 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(122): Constructor flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000, compactMin=4 2023-04-25 02:53:40,791 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=5 2023-04-25 02:53:40,791 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(562): Starting 5 core workers (bigger of cpus/4 or 16) with max (burst) worker count=50 2023-04-25 02:53:40,794 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-04-25 02:53:40,796 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 1 msec 2023-04-25 02:53:40,829 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 32 msec 2023-04-25 02:53:40,829 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.RemoteProcedureDispatcher(95): Instantiated, coreThreads=3 (allowCoreThreadTimeOut=true), queueMaxSize=32, operationDelay=150 2023-04-25 02:53:40,896 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(253): hbase:meta replica znodes: [] 2023-04-25 02:53:40,902 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.RegionServerTracker(124): Starting RegionServerTracker; 0 have existing ServerCrashProcedures, 0 possibly 'live' servers, and 0 'splitting'. 2023-04-25 02:53:40,927 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.BaseLoadBalancer(1082): slop=0.001, systemTablesOnMaster=false 2023-04-25 02:53:40,931 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, ServerLocalityCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-04-25 02:53:40,933 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2023-04-25 02:53:40,938 INFO [master/jenkins-hbase4:0:becomeActiveMaster] normalizer.RegionNormalizerWorker(118): Normalizer rate limit set to unlimited 2023-04-25 02:53:40,942 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2023-04-25 02:53:41,055 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-25 02:53:41,058 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2023-04-25 02:53:41,059 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2023-04-25 02:53:41,075 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/snapshot-cleanup 2023-04-25 02:53:41,132 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-04-25 02:53:41,132 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-04-25 02:53:41,133 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-25 02:53:41,133 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(744): Active/primary master=jenkins-hbase4.apache.org,44265,1682391218835, sessionid=0x101b93ca1070000, setting cluster-up flag (Was=false) 2023-04-25 02:53:41,170 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-25 02:53:41,193 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/flush-table-proc/acquired, /hbase/flush-table-proc/reached, /hbase/flush-table-proc/abort 2023-04-25 02:53:41,195 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,44265,1682391218835 2023-04-25 02:53:41,209 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-25 02:53:41,232 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/online-snapshot/acquired, /hbase/online-snapshot/reached, /hbase/online-snapshot/abort 2023-04-25 02:53:41,234 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,44265,1682391218835 2023-04-25 02:53:41,236 WARN [master/jenkins-hbase4:0:becomeActiveMaster] snapshot.SnapshotManager(302): Couldn't delete working snapshot directory: hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/.hbase-snapshot/.tmp 2023-04-25 02:53:41,262 INFO [RS:0;jenkins-hbase4:46697] regionserver.HRegionServer(952): ClusterId : 42edb776-2c8f-44f4-8800-1527a17d0d3b 2023-04-25 02:53:41,267 DEBUG [RS:0;jenkins-hbase4:46697] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-04-25 02:53:41,273 DEBUG [RS:0;jenkins-hbase4:46697] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-04-25 02:53:41,273 DEBUG [RS:0;jenkins-hbase4:46697] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-04-25 02:53:41,278 DEBUG [RS:0;jenkins-hbase4:46697] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-04-25 02:53:41,279 DEBUG [RS:0;jenkins-hbase4:46697] zookeeper.ReadOnlyZKClient(139): Connect 0x30bfadd0 to 127.0.0.1:62054 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-25 02:53:41,284 DEBUG [RS:0;jenkins-hbase4:46697] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3e7855b7, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-25 02:53:41,285 DEBUG [RS:0;jenkins-hbase4:46697] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3a47c6a0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-04-25 02:53:41,316 DEBUG [RS:0;jenkins-hbase4:46697] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:0;jenkins-hbase4:46697 2023-04-25 02:53:41,321 INFO [RS:0;jenkins-hbase4:46697] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-04-25 02:53:41,322 INFO [RS:0;jenkins-hbase4:46697] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-04-25 02:53:41,322 DEBUG [RS:0;jenkins-hbase4:46697] regionserver.HRegionServer(1023): About to register with Master. 2023-04-25 02:53:41,325 INFO [RS:0;jenkins-hbase4:46697] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase4.apache.org,44265,1682391218835 with isa=jenkins-hbase4.apache.org/172.31.14.131:46697, startcode=1682391220023 2023-04-25 02:53:41,348 DEBUG [RS:0;jenkins-hbase4:46697] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-04-25 02:53:41,368 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT; InitMetaProcedure table=hbase:meta 2023-04-25 02:53:41,379 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_OPEN_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-04-25 02:53:41,379 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_CLOSE_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-04-25 02:53:41,379 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-04-25 02:53:41,380 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_META_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-04-25 02:53:41,380 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=M_LOG_REPLAY_OPS-master/jenkins-hbase4:0, corePoolSize=10, maxPoolSize=10 2023-04-25 02:53:41,380 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-25 02:53:41,380 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_MERGE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-04-25 02:53:41,380 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_TABLE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-25 02:53:41,389 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1682391251388 2023-04-25 02:53:41,391 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): log_cleaner Cleaner pool size is 1 2023-04-25 02:53:41,394 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT, locked=true; InitMetaProcedure table=hbase:meta 2023-04-25 02:53:41,394 INFO [PEWorker-1] procedure.InitMetaProcedure(71): BOOTSTRAP: creating hbase:meta region 2023-04-25 02:53:41,400 INFO [PEWorker-1] util.FSTableDescriptors(128): Creating new hbase:meta table descriptor 'hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-04-25 02:53:41,404 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2023-04-25 02:53:41,413 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2023-04-25 02:53:41,413 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreWALCleaner 2023-04-25 02:53:41,414 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:38953, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins.hfs.0 (auth:SIMPLE), service=RegionServerStatusService 2023-04-25 02:53:41,414 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2023-04-25 02:53:41,415 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.LogCleaner(148): Creating 1 old WALs cleaner threads 2023-04-25 02:53:41,416 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=LogsCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-04-25 02:53:41,417 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): hfile_cleaner Cleaner pool size is 2 2023-04-25 02:53:41,419 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreHFileCleaner 2023-04-25 02:53:41,419 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2023-04-25 02:53:41,424 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2023-04-25 02:53:41,425 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2023-04-25 02:53:41,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2832) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerStartup(MasterRpcServices.java:577) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15952) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:53:41,431 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(242): Starting for large file=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1682391221428,5,FailOnTimeoutGroup] 2023-04-25 02:53:41,436 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(257): Starting for small files=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1682391221433,5,FailOnTimeoutGroup] 2023-04-25 02:53:41,436 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HFileCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-04-25 02:53:41,437 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1461): Reopening regions with very high storeFileRefCount is disabled. Provide threshold value > 0 for hbase.regions.recovery.store.file.ref.count to enable it. 2023-04-25 02:53:41,440 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=ReplicationBarrierCleaner, period=43200000, unit=MILLISECONDS is enabled. 2023-04-25 02:53:41,441 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=SnapshotCleaner, period=1800000, unit=MILLISECONDS is enabled. 2023-04-25 02:53:41,449 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-04-25 02:53:41,450 INFO [PEWorker-1] util.FSTableDescriptors(135): Updated hbase:meta table descriptor to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-04-25 02:53:41,451 INFO [PEWorker-1] regionserver.HRegion(7675): creating {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f 2023-04-25 02:53:41,463 DEBUG [RS:0;jenkins-hbase4:46697] regionserver.HRegionServer(2829): Master is not running yet 2023-04-25 02:53:41,463 WARN [RS:0;jenkins-hbase4:46697] regionserver.HRegionServer(1031): reportForDuty failed; sleeping 100 ms and then retrying. 2023-04-25 02:53:41,565 INFO [RS:0;jenkins-hbase4:46697] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase4.apache.org,44265,1682391218835 with isa=jenkins-hbase4.apache.org/172.31.14.131:46697, startcode=1682391220023 2023-04-25 02:53:41,572 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:41,579 DEBUG [RS:0;jenkins-hbase4:46697] regionserver.HRegionServer(1596): Config from master: hbase.rootdir=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f 2023-04-25 02:53:41,579 DEBUG [RS:0;jenkins-hbase4:46697] regionserver.HRegionServer(1596): Config from master: fs.defaultFS=hdfs://localhost:43711 2023-04-25 02:53:41,580 DEBUG [RS:0;jenkins-hbase4:46697] regionserver.HRegionServer(1596): Config from master: hbase.master.info.port=-1 2023-04-25 02:53:41,586 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-04-25 02:53:41,587 DEBUG [RS:0;jenkins-hbase4:46697] zookeeper.ZKUtil(162): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:41,587 WARN [RS:0;jenkins-hbase4:46697] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-04-25 02:53:41,588 INFO [RS:0;jenkins-hbase4:46697] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-04-25 02:53:41,588 DEBUG [RS:0;jenkins-hbase4:46697] regionserver.HRegionServer(1947): logDir=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/WALs/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:41,590 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,46697,1682391220023] 2023-04-25 02:53:41,599 DEBUG [RS:0;jenkins-hbase4:46697] zookeeper.ZKUtil(162): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:41,610 DEBUG [RS:0;jenkins-hbase4:46697] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-04-25 02:53:41,619 INFO [RS:0;jenkins-hbase4:46697] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-04-25 02:53:41,644 INFO [RS:0;jenkins-hbase4:46697] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-04-25 02:53:41,649 INFO [RS:0;jenkins-hbase4:46697] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-04-25 02:53:41,649 INFO [RS:0;jenkins-hbase4:46697] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-04-25 02:53:41,651 INFO [RS:0;jenkins-hbase4:46697] regionserver.HRegionServer$CompactionChecker(1838): CompactionChecker runs every PT1S 2023-04-25 02:53:41,660 INFO [RS:0;jenkins-hbase4:46697] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-04-25 02:53:41,660 DEBUG [RS:0;jenkins-hbase4:46697] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-25 02:53:41,660 DEBUG [RS:0;jenkins-hbase4:46697] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-25 02:53:41,660 DEBUG [RS:0;jenkins-hbase4:46697] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-25 02:53:41,661 DEBUG [RS:0;jenkins-hbase4:46697] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-25 02:53:41,661 DEBUG [RS:0;jenkins-hbase4:46697] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-25 02:53:41,661 DEBUG [RS:0;jenkins-hbase4:46697] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-04-25 02:53:41,661 DEBUG [RS:0;jenkins-hbase4:46697] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-25 02:53:41,661 DEBUG [RS:0;jenkins-hbase4:46697] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-25 02:53:41,661 DEBUG [RS:0;jenkins-hbase4:46697] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-25 02:53:41,662 DEBUG [RS:0;jenkins-hbase4:46697] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-25 02:53:41,663 INFO [RS:0;jenkins-hbase4:46697] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-04-25 02:53:41,663 INFO [RS:0;jenkins-hbase4:46697] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-04-25 02:53:41,663 INFO [RS:0;jenkins-hbase4:46697] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-04-25 02:53:41,682 INFO [RS:0;jenkins-hbase4:46697] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-04-25 02:53:41,684 INFO [RS:0;jenkins-hbase4:46697] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,46697,1682391220023-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-04-25 02:53:41,703 INFO [RS:0;jenkins-hbase4:46697] regionserver.Replication(203): jenkins-hbase4.apache.org,46697,1682391220023 started 2023-04-25 02:53:41,703 INFO [RS:0;jenkins-hbase4:46697] regionserver.HRegionServer(1638): Serving as jenkins-hbase4.apache.org,46697,1682391220023, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:46697, sessionid=0x101b93ca1070001 2023-04-25 02:53:41,703 DEBUG [RS:0;jenkins-hbase4:46697] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-04-25 02:53:41,703 DEBUG [RS:0;jenkins-hbase4:46697] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:41,703 DEBUG [RS:0;jenkins-hbase4:46697] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,46697,1682391220023' 2023-04-25 02:53:41,703 DEBUG [RS:0;jenkins-hbase4:46697] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:53:41,704 DEBUG [RS:0;jenkins-hbase4:46697] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:53:41,705 DEBUG [RS:0;jenkins-hbase4:46697] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-04-25 02:53:41,705 DEBUG [RS:0;jenkins-hbase4:46697] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-04-25 02:53:41,705 DEBUG [RS:0;jenkins-hbase4:46697] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:41,705 DEBUG [RS:0;jenkins-hbase4:46697] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,46697,1682391220023' 2023-04-25 02:53:41,705 DEBUG [RS:0;jenkins-hbase4:46697] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-04-25 02:53:41,705 DEBUG [RS:0;jenkins-hbase4:46697] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-04-25 02:53:41,706 DEBUG [RS:0;jenkins-hbase4:46697] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-04-25 02:53:41,706 INFO [RS:0;jenkins-hbase4:46697] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-04-25 02:53:41,706 INFO [RS:0;jenkins-hbase4:46697] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-04-25 02:53:41,819 INFO [RS:0;jenkins-hbase4:46697] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C46697%2C1682391220023, suffix=, logDir=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/WALs/jenkins-hbase4.apache.org,46697,1682391220023, archiveDir=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/oldWALs, maxLogs=32 2023-04-25 02:53:41,839 DEBUG [RS-EventLoopGroup-3-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:35477,DS-db6f9fb0-c5d9-4913-bf97-d523392806fc,DISK] 2023-04-25 02:53:41,843 INFO [RS:0;jenkins-hbase4:46697] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/WALs/jenkins-hbase4.apache.org,46697,1682391220023/jenkins-hbase4.apache.org%2C46697%2C1682391220023.1682391221821 2023-04-25 02:53:41,843 DEBUG [RS:0;jenkins-hbase4:46697] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:35477,DS-db6f9fb0-c5d9-4913-bf97-d523392806fc,DISK]] 2023-04-25 02:53:41,874 DEBUG [PEWorker-1] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-25 02:53:41,876 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-04-25 02:53:41,879 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/hbase/meta/1588230740/info 2023-04-25 02:53:41,880 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-04-25 02:53:41,881 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-25 02:53:41,881 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-04-25 02:53:41,883 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/hbase/meta/1588230740/rep_barrier 2023-04-25 02:53:41,884 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-04-25 02:53:41,886 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-25 02:53:41,886 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-04-25 02:53:41,888 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/hbase/meta/1588230740/table 2023-04-25 02:53:41,888 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-04-25 02:53:41,889 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-25 02:53:41,891 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/hbase/meta/1588230740 2023-04-25 02:53:41,891 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/hbase/meta/1588230740 2023-04-25 02:53:41,895 DEBUG [PEWorker-1] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-25 02:53:41,897 DEBUG [PEWorker-1] regionserver.HRegion(1055): writing seq id for 1588230740 2023-04-25 02:53:41,901 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/hbase/meta/1588230740/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-25 02:53:41,902 INFO [PEWorker-1] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=74942367, jitterRate=0.1167282909154892}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-25 02:53:41,902 DEBUG [PEWorker-1] regionserver.HRegion(965): Region open journal for 1588230740: 2023-04-25 02:53:41,902 DEBUG [PEWorker-1] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-04-25 02:53:41,902 INFO [PEWorker-1] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-04-25 02:53:41,902 DEBUG [PEWorker-1] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-04-25 02:53:41,902 DEBUG [PEWorker-1] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-04-25 02:53:41,902 DEBUG [PEWorker-1] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-04-25 02:53:41,904 INFO [PEWorker-1] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-04-25 02:53:41,904 DEBUG [PEWorker-1] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-04-25 02:53:41,908 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_ASSIGN_META, locked=true; InitMetaProcedure table=hbase:meta 2023-04-25 02:53:41,908 INFO [PEWorker-1] procedure.InitMetaProcedure(103): Going to assign meta 2023-04-25 02:53:41,915 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN}] 2023-04-25 02:53:41,926 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN 2023-04-25 02:53:41,929 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; state=OFFLINE, location=null; forceNewPlan=false, retain=false 2023-04-25 02:53:42,081 DEBUG [jenkins-hbase4:44265] assignment.AssignmentManager(2176): Processing assignQueue; systemServersCount=1, allServersCount=1 2023-04-25 02:53:42,084 INFO [PEWorker-4] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,46697,1682391220023, state=OPENING 2023-04-25 02:53:42,090 DEBUG [PEWorker-4] zookeeper.MetaTableLocator(240): hbase:meta region location doesn't exist, create it 2023-04-25 02:53:42,093 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-25 02:53:42,094 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-04-25 02:53:42,096 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=3, ppid=2, state=RUNNABLE; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,46697,1682391220023}] 2023-04-25 02:53:42,282 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(712): New admin connection to jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:42,285 DEBUG [RSProcedureDispatcher-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-04-25 02:53:42,289 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:36460, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-04-25 02:53:42,300 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open hbase:meta,,1.1588230740 2023-04-25 02:53:42,300 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-04-25 02:53:42,304 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C46697%2C1682391220023.meta, suffix=.meta, logDir=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/WALs/jenkins-hbase4.apache.org,46697,1682391220023, archiveDir=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/oldWALs, maxLogs=32 2023-04-25 02:53:42,321 DEBUG [RS-EventLoopGroup-3-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:35477,DS-db6f9fb0-c5d9-4913-bf97-d523392806fc,DISK] 2023-04-25 02:53:42,327 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/WALs/jenkins-hbase4.apache.org,46697,1682391220023/jenkins-hbase4.apache.org%2C46697%2C1682391220023.meta.1682391222305.meta 2023-04-25 02:53:42,328 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:35477,DS-db6f9fb0-c5d9-4913-bf97-d523392806fc,DISK]] 2023-04-25 02:53:42,328 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2023-04-25 02:53:42,330 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-04-25 02:53:42,350 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(8550): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2023-04-25 02:53:42,357 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2023-04-25 02:53:42,363 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table meta 1588230740 2023-04-25 02:53:42,363 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-25 02:53:42,363 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 1588230740 2023-04-25 02:53:42,364 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 1588230740 2023-04-25 02:53:42,367 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-04-25 02:53:42,369 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/hbase/meta/1588230740/info 2023-04-25 02:53:42,369 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/hbase/meta/1588230740/info 2023-04-25 02:53:42,369 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-04-25 02:53:42,370 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-25 02:53:42,370 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-04-25 02:53:42,372 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/hbase/meta/1588230740/rep_barrier 2023-04-25 02:53:42,372 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/hbase/meta/1588230740/rep_barrier 2023-04-25 02:53:42,373 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-04-25 02:53:42,374 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-25 02:53:42,374 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-04-25 02:53:42,376 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/hbase/meta/1588230740/table 2023-04-25 02:53:42,376 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/hbase/meta/1588230740/table 2023-04-25 02:53:42,377 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-04-25 02:53:42,378 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-25 02:53:42,379 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/hbase/meta/1588230740 2023-04-25 02:53:42,382 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/hbase/meta/1588230740 2023-04-25 02:53:42,385 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-25 02:53:42,387 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 1588230740 2023-04-25 02:53:42,389 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=63051190, jitterRate=-0.060464054346084595}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-25 02:53:42,389 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 1588230740: 2023-04-25 02:53:42,399 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:meta,,1.1588230740, pid=3, masterSystemTime=1682391222273 2023-04-25 02:53:42,417 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:meta,,1.1588230740 2023-04-25 02:53:42,418 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened hbase:meta,,1.1588230740 2023-04-25 02:53:42,418 INFO [PEWorker-5] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,46697,1682391220023, state=OPEN 2023-04-25 02:53:42,421 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-04-25 02:53:42,421 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-04-25 02:53:42,426 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=3, resume processing ppid=2 2023-04-25 02:53:42,426 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=3, ppid=2, state=SUCCESS; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,46697,1682391220023 in 325 msec 2023-04-25 02:53:42,431 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=2, resume processing ppid=1 2023-04-25 02:53:42,431 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=2, ppid=1, state=SUCCESS; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN in 512 msec 2023-04-25 02:53:42,437 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta in 1.1490 sec 2023-04-25 02:53:42,437 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(953): Wait for region servers to report in: status=null, state=RUNNING, startTime=1682391222437, completionTime=-1 2023-04-25 02:53:42,437 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ServerManager(821): Finished waiting on RegionServer count=1; waited=0ms, expected min=1 server(s), max=1 server(s), master is running 2023-04-25 02:53:42,438 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1517): Joining cluster... 2023-04-25 02:53:42,508 DEBUG [hconnection-0x14e0c1f2-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-25 02:53:42,512 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:36464, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-25 02:53:42,529 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1529): Number of RegionServers=1 2023-04-25 02:53:42,529 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1682391282529 2023-04-25 02:53:42,529 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1682391342529 2023-04-25 02:53:42,530 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1536): Joined the cluster in 91 msec 2023-04-25 02:53:42,552 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,44265,1682391218835-ClusterStatusChore, period=60000, unit=MILLISECONDS is enabled. 2023-04-25 02:53:42,552 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,44265,1682391218835-BalancerChore, period=300000, unit=MILLISECONDS is enabled. 2023-04-25 02:53:42,552 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,44265,1682391218835-RegionNormalizerChore, period=300000, unit=MILLISECONDS is enabled. 2023-04-25 02:53:42,554 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=CatalogJanitor-jenkins-hbase4:44265, period=300000, unit=MILLISECONDS is enabled. 2023-04-25 02:53:42,555 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HbckChore-, period=3600000, unit=MILLISECONDS is enabled. 2023-04-25 02:53:42,561 DEBUG [master/jenkins-hbase4:0.Chore.1] janitor.CatalogJanitor(175): 2023-04-25 02:53:42,570 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.TableNamespaceManager(92): Namespace table not found. Creating... 2023-04-25 02:53:42,571 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2148): Client=null/null create 'hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-04-25 02:53:42,581 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=hbase:namespace 2023-04-25 02:53:42,586 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_PRE_OPERATION 2023-04-25 02:53:42,589 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-25 02:53:42,614 DEBUG [HFileArchiver-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/.tmp/data/hbase/namespace/4c9864611c8b9dc201fb679c2744e7a2 2023-04-25 02:53:42,617 DEBUG [HFileArchiver-1] backup.HFileArchiver(153): Directory hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/.tmp/data/hbase/namespace/4c9864611c8b9dc201fb679c2744e7a2 empty. 2023-04-25 02:53:42,618 DEBUG [HFileArchiver-1] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/.tmp/data/hbase/namespace/4c9864611c8b9dc201fb679c2744e7a2 2023-04-25 02:53:42,618 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(328): Archived hbase:namespace regions 2023-04-25 02:53:43,045 DEBUG [PEWorker-4] util.FSTableDescriptors(570): Wrote into hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/.tmp/data/hbase/namespace/.tabledesc/.tableinfo.0000000001 2023-04-25 02:53:43,047 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(7675): creating {ENCODED => 4c9864611c8b9dc201fb679c2744e7a2, NAME => 'hbase:namespace,,1682391222571.4c9864611c8b9dc201fb679c2744e7a2.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/.tmp 2023-04-25 02:53:43,462 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1682391222571.4c9864611c8b9dc201fb679c2744e7a2.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-25 02:53:43,462 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1604): Closing 4c9864611c8b9dc201fb679c2744e7a2, disabling compactions & flushes 2023-04-25 02:53:43,462 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1682391222571.4c9864611c8b9dc201fb679c2744e7a2. 2023-04-25 02:53:43,463 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1682391222571.4c9864611c8b9dc201fb679c2744e7a2. 2023-04-25 02:53:43,463 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1682391222571.4c9864611c8b9dc201fb679c2744e7a2. after waiting 0 ms 2023-04-25 02:53:43,463 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1682391222571.4c9864611c8b9dc201fb679c2744e7a2. 2023-04-25 02:53:43,463 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1838): Closed hbase:namespace,,1682391222571.4c9864611c8b9dc201fb679c2744e7a2. 2023-04-25 02:53:43,463 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1558): Region close journal for 4c9864611c8b9dc201fb679c2744e7a2: 2023-04-25 02:53:43,468 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ADD_TO_META 2023-04-25 02:53:43,483 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"hbase:namespace,,1682391222571.4c9864611c8b9dc201fb679c2744e7a2.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1682391223471"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682391223471"}]},"ts":"1682391223471"} 2023-04-25 02:53:43,510 INFO [PEWorker-4] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-25 02:53:43,512 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-25 02:53:43,516 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682391223512"}]},"ts":"1682391223512"} 2023-04-25 02:53:43,521 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLING in hbase:meta 2023-04-25 02:53:43,534 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=4c9864611c8b9dc201fb679c2744e7a2, ASSIGN}] 2023-04-25 02:53:43,536 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=4c9864611c8b9dc201fb679c2744e7a2, ASSIGN 2023-04-25 02:53:43,538 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:namespace, region=4c9864611c8b9dc201fb679c2744e7a2, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,46697,1682391220023; forceNewPlan=false, retain=false 2023-04-25 02:53:43,689 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=4c9864611c8b9dc201fb679c2744e7a2, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:43,689 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:namespace,,1682391222571.4c9864611c8b9dc201fb679c2744e7a2.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1682391223688"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682391223688"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682391223688"}]},"ts":"1682391223688"} 2023-04-25 02:53:43,695 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=6, ppid=5, state=RUNNABLE; OpenRegionProcedure 4c9864611c8b9dc201fb679c2744e7a2, server=jenkins-hbase4.apache.org,46697,1682391220023}] 2023-04-25 02:53:43,855 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open hbase:namespace,,1682391222571.4c9864611c8b9dc201fb679c2744e7a2. 2023-04-25 02:53:43,855 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 4c9864611c8b9dc201fb679c2744e7a2, NAME => 'hbase:namespace,,1682391222571.4c9864611c8b9dc201fb679c2744e7a2.', STARTKEY => '', ENDKEY => ''} 2023-04-25 02:53:43,856 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table namespace 4c9864611c8b9dc201fb679c2744e7a2 2023-04-25 02:53:43,856 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1682391222571.4c9864611c8b9dc201fb679c2744e7a2.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-25 02:53:43,857 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 4c9864611c8b9dc201fb679c2744e7a2 2023-04-25 02:53:43,857 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 4c9864611c8b9dc201fb679c2744e7a2 2023-04-25 02:53:43,859 INFO [StoreOpener-4c9864611c8b9dc201fb679c2744e7a2-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 4c9864611c8b9dc201fb679c2744e7a2 2023-04-25 02:53:43,861 DEBUG [StoreOpener-4c9864611c8b9dc201fb679c2744e7a2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/hbase/namespace/4c9864611c8b9dc201fb679c2744e7a2/info 2023-04-25 02:53:43,861 DEBUG [StoreOpener-4c9864611c8b9dc201fb679c2744e7a2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/hbase/namespace/4c9864611c8b9dc201fb679c2744e7a2/info 2023-04-25 02:53:43,862 INFO [StoreOpener-4c9864611c8b9dc201fb679c2744e7a2-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 4c9864611c8b9dc201fb679c2744e7a2 columnFamilyName info 2023-04-25 02:53:43,862 INFO [StoreOpener-4c9864611c8b9dc201fb679c2744e7a2-1] regionserver.HStore(310): Store=4c9864611c8b9dc201fb679c2744e7a2/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-25 02:53:43,864 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/hbase/namespace/4c9864611c8b9dc201fb679c2744e7a2 2023-04-25 02:53:43,865 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/hbase/namespace/4c9864611c8b9dc201fb679c2744e7a2 2023-04-25 02:53:43,870 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 4c9864611c8b9dc201fb679c2744e7a2 2023-04-25 02:53:43,874 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/hbase/namespace/4c9864611c8b9dc201fb679c2744e7a2/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-25 02:53:43,874 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 4c9864611c8b9dc201fb679c2744e7a2; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=60639701, jitterRate=-0.09639804065227509}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-04-25 02:53:43,875 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 4c9864611c8b9dc201fb679c2744e7a2: 2023-04-25 02:53:43,877 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:namespace,,1682391222571.4c9864611c8b9dc201fb679c2744e7a2., pid=6, masterSystemTime=1682391223849 2023-04-25 02:53:43,882 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:namespace,,1682391222571.4c9864611c8b9dc201fb679c2744e7a2. 2023-04-25 02:53:43,882 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened hbase:namespace,,1682391222571.4c9864611c8b9dc201fb679c2744e7a2. 2023-04-25 02:53:43,883 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=4c9864611c8b9dc201fb679c2744e7a2, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:43,884 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:namespace,,1682391222571.4c9864611c8b9dc201fb679c2744e7a2.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1682391223883"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1682391223883"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1682391223883"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1682391223883"}]},"ts":"1682391223883"} 2023-04-25 02:53:43,893 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=6, resume processing ppid=5 2023-04-25 02:53:43,893 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=6, ppid=5, state=SUCCESS; OpenRegionProcedure 4c9864611c8b9dc201fb679c2744e7a2, server=jenkins-hbase4.apache.org,46697,1682391220023 in 194 msec 2023-04-25 02:53:43,896 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=5, resume processing ppid=4 2023-04-25 02:53:43,897 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=5, ppid=4, state=SUCCESS; TransitRegionStateProcedure table=hbase:namespace, region=4c9864611c8b9dc201fb679c2744e7a2, ASSIGN in 359 msec 2023-04-25 02:53:43,898 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-25 02:53:43,898 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682391223898"}]},"ts":"1682391223898"} 2023-04-25 02:53:43,901 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLED in hbase:meta 2023-04-25 02:53:43,905 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_POST_OPERATION 2023-04-25 02:53:43,908 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace in 1.3330 sec 2023-04-25 02:53:43,985 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/namespace 2023-04-25 02:53:43,986 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/namespace 2023-04-25 02:53:43,986 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-25 02:53:44,030 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=7, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=default 2023-04-25 02:53:44,051 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-04-25 02:53:44,057 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default in 37 msec 2023-04-25 02:53:44,063 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=8, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=hbase 2023-04-25 02:53:44,074 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-04-25 02:53:44,079 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase in 15 msec 2023-04-25 02:53:44,092 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2023-04-25 02:53:44,094 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2023-04-25 02:53:44,095 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1083): Master has completed initialization 3.988sec 2023-04-25 02:53:44,097 INFO [master/jenkins-hbase4:0:becomeActiveMaster] quotas.MasterQuotaManager(97): Quota support disabled 2023-04-25 02:53:44,099 INFO [master/jenkins-hbase4:0:becomeActiveMaster] slowlog.SlowLogMasterService(57): Slow/Large requests logging to system table hbase:slowlog is disabled. Quitting. 2023-04-25 02:53:44,099 INFO [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKWatcher(269): not a secure deployment, proceeding 2023-04-25 02:53:44,100 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,44265,1682391218835-ExpiredMobFileCleanerChore, period=86400, unit=SECONDS is enabled. 2023-04-25 02:53:44,101 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,44265,1682391218835-MobCompactionChore, period=604800, unit=SECONDS is enabled. 2023-04-25 02:53:44,114 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1175): Balancer post startup initialization complete, took 0 seconds 2023-04-25 02:53:44,170 DEBUG [Listener at localhost/41915] zookeeper.ReadOnlyZKClient(139): Connect 0x0f3002be to 127.0.0.1:62054 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-25 02:53:44,176 DEBUG [Listener at localhost/41915] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5e324982, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-25 02:53:44,190 DEBUG [hconnection-0x10a3ae0a-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-25 02:53:44,203 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:43776, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-25 02:53:44,214 INFO [Listener at localhost/41915] hbase.HBaseTestingUtility(1145): Minicluster is up; activeMaster=jenkins-hbase4.apache.org,44265,1682391218835 2023-04-25 02:53:44,235 INFO [Listener at localhost/41915] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMixedAtomicity Thread=206, OpenFileDescriptor=470, MaxFileDescriptor=60000, SystemLoadAverage=409, ProcessCount=170, AvailableMemoryMB=5108 2023-04-25 02:53:44,251 DEBUG [Listener at localhost/41915] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-25 02:53:44,255 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:49028, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-25 02:53:44,263 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-25 02:53:44,267 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-25 02:53:44,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure2.ProcedureExecutor(1029): Stored pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-25 02:53:44,271 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-25 02:53:44,273 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-25 02:53:44,276 DEBUG [HFileArchiver-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/.tmp/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a 2023-04-25 02:53:44,277 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 9 2023-04-25 02:53:44,278 DEBUG [HFileArchiver-2] backup.HFileArchiver(153): Directory hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/.tmp/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a empty. 2023-04-25 02:53:44,280 DEBUG [HFileArchiver-2] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/.tmp/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a 2023-04-25 02:53:44,280 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-25 02:53:44,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-25 02:53:44,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-25 02:53:44,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-25 02:53:44,700 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-25 02:53:44,702 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 047cf722b8f6bff032073f994134764a, NAME => 'TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/.tmp 2023-04-25 02:53:44,714 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-25 02:53:44,714 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 047cf722b8f6bff032073f994134764a, disabling compactions & flushes 2023-04-25 02:53:44,714 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. 2023-04-25 02:53:44,714 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. 2023-04-25 02:53:44,714 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. after waiting 0 ms 2023-04-25 02:53:44,714 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. 2023-04-25 02:53:44,715 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. 2023-04-25 02:53:44,715 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 047cf722b8f6bff032073f994134764a: 2023-04-25 02:53:44,719 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-25 02:53:44,721 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682391224721"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682391224721"}]},"ts":"1682391224721"} 2023-04-25 02:53:44,724 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-25 02:53:44,725 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-25 02:53:44,725 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682391224725"}]},"ts":"1682391224725"} 2023-04-25 02:53:44,727 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-25 02:53:44,731 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=047cf722b8f6bff032073f994134764a, ASSIGN}] 2023-04-25 02:53:44,733 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=047cf722b8f6bff032073f994134764a, ASSIGN 2023-04-25 02:53:44,734 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=047cf722b8f6bff032073f994134764a, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,46697,1682391220023; forceNewPlan=false, retain=false 2023-04-25 02:53:44,885 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=047cf722b8f6bff032073f994134764a, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:44,886 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682391224885"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682391224885"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682391224885"}]},"ts":"1682391224885"} 2023-04-25 02:53:44,890 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=11, ppid=10, state=RUNNABLE; OpenRegionProcedure 047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023}] 2023-04-25 02:53:44,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-25 02:53:45,051 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. 2023-04-25 02:53:45,051 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 047cf722b8f6bff032073f994134764a, NAME => 'TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a.', STARTKEY => '', ENDKEY => ''} 2023-04-25 02:53:45,052 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 047cf722b8f6bff032073f994134764a 2023-04-25 02:53:45,052 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-25 02:53:45,052 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 047cf722b8f6bff032073f994134764a 2023-04-25 02:53:45,052 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 047cf722b8f6bff032073f994134764a 2023-04-25 02:53:45,055 INFO [StoreOpener-047cf722b8f6bff032073f994134764a-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 047cf722b8f6bff032073f994134764a 2023-04-25 02:53:45,057 DEBUG [StoreOpener-047cf722b8f6bff032073f994134764a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/A 2023-04-25 02:53:45,057 DEBUG [StoreOpener-047cf722b8f6bff032073f994134764a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/A 2023-04-25 02:53:45,059 INFO [StoreOpener-047cf722b8f6bff032073f994134764a-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-25 02:53:45,059 INFO [StoreOpener-047cf722b8f6bff032073f994134764a-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 047cf722b8f6bff032073f994134764a columnFamilyName A 2023-04-25 02:53:45,060 INFO [StoreOpener-047cf722b8f6bff032073f994134764a-1] regionserver.HStore(310): Store=047cf722b8f6bff032073f994134764a/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-25 02:53:45,060 INFO [StoreOpener-047cf722b8f6bff032073f994134764a-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 047cf722b8f6bff032073f994134764a 2023-04-25 02:53:45,064 DEBUG [StoreOpener-047cf722b8f6bff032073f994134764a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/B 2023-04-25 02:53:45,064 DEBUG [StoreOpener-047cf722b8f6bff032073f994134764a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/B 2023-04-25 02:53:45,065 INFO [StoreOpener-047cf722b8f6bff032073f994134764a-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-25 02:53:45,065 INFO [StoreOpener-047cf722b8f6bff032073f994134764a-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 047cf722b8f6bff032073f994134764a columnFamilyName B 2023-04-25 02:53:45,066 INFO [StoreOpener-047cf722b8f6bff032073f994134764a-1] regionserver.HStore(310): Store=047cf722b8f6bff032073f994134764a/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-25 02:53:45,067 INFO [StoreOpener-047cf722b8f6bff032073f994134764a-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 047cf722b8f6bff032073f994134764a 2023-04-25 02:53:45,069 DEBUG [StoreOpener-047cf722b8f6bff032073f994134764a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/C 2023-04-25 02:53:45,069 DEBUG [StoreOpener-047cf722b8f6bff032073f994134764a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/C 2023-04-25 02:53:45,069 INFO [StoreOpener-047cf722b8f6bff032073f994134764a-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-25 02:53:45,070 INFO [StoreOpener-047cf722b8f6bff032073f994134764a-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 047cf722b8f6bff032073f994134764a columnFamilyName C 2023-04-25 02:53:45,070 INFO [StoreOpener-047cf722b8f6bff032073f994134764a-1] regionserver.HStore(310): Store=047cf722b8f6bff032073f994134764a/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-25 02:53:45,071 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. 2023-04-25 02:53:45,073 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a 2023-04-25 02:53:45,074 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a 2023-04-25 02:53:45,078 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-25 02:53:45,081 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 047cf722b8f6bff032073f994134764a 2023-04-25 02:53:45,084 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-25 02:53:45,085 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 047cf722b8f6bff032073f994134764a; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=62589715, jitterRate=-0.06734056770801544}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-25 02:53:45,085 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 047cf722b8f6bff032073f994134764a: 2023-04-25 02:53:45,087 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a., pid=11, masterSystemTime=1682391225044 2023-04-25 02:53:45,090 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. 2023-04-25 02:53:45,090 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. 2023-04-25 02:53:45,091 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=047cf722b8f6bff032073f994134764a, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:45,091 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682391225090"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1682391225090"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1682391225090"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1682391225090"}]},"ts":"1682391225090"} 2023-04-25 02:53:45,097 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=11, resume processing ppid=10 2023-04-25 02:53:45,097 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=11, ppid=10, state=SUCCESS; OpenRegionProcedure 047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 in 204 msec 2023-04-25 02:53:45,101 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=10, resume processing ppid=9 2023-04-25 02:53:45,101 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=10, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=047cf722b8f6bff032073f994134764a, ASSIGN in 366 msec 2023-04-25 02:53:45,103 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-25 02:53:45,103 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682391225103"}]},"ts":"1682391225103"} 2023-04-25 02:53:45,105 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-25 02:53:45,109 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-25 02:53:45,112 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=9, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 842 msec 2023-04-25 02:53:45,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-25 02:53:45,405 INFO [Listener at localhost/41915] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 9 completed 2023-04-25 02:53:45,410 DEBUG [Listener at localhost/41915] zookeeper.ReadOnlyZKClient(139): Connect 0x08c3f619 to 127.0.0.1:62054 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-25 02:53:45,417 DEBUG [Listener at localhost/41915] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@b17e1c1, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-25 02:53:45,419 DEBUG [Listener at localhost/41915] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-25 02:53:45,423 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:43788, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-25 02:53:45,427 DEBUG [Listener at localhost/41915] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-25 02:53:45,429 INFO [RS-EventLoopGroup-1-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:49036, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-25 02:53:45,438 DEBUG [Listener at localhost/41915] zookeeper.ReadOnlyZKClient(139): Connect 0x748f869b to 127.0.0.1:62054 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-25 02:53:45,443 DEBUG [Listener at localhost/41915] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@56c8e45e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-25 02:53:45,443 DEBUG [Listener at localhost/41915] zookeeper.ReadOnlyZKClient(139): Connect 0x56223d42 to 127.0.0.1:62054 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-25 02:53:45,448 DEBUG [Listener at localhost/41915] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4fea27db, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-25 02:53:45,449 DEBUG [Listener at localhost/41915] zookeeper.ReadOnlyZKClient(139): Connect 0x30eb7fb2 to 127.0.0.1:62054 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-25 02:53:45,452 DEBUG [Listener at localhost/41915] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6038a026, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-25 02:53:45,453 DEBUG [Listener at localhost/41915] zookeeper.ReadOnlyZKClient(139): Connect 0x2de4d1f6 to 127.0.0.1:62054 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-25 02:53:45,458 DEBUG [Listener at localhost/41915] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@63859c62, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-25 02:53:45,458 DEBUG [Listener at localhost/41915] zookeeper.ReadOnlyZKClient(139): Connect 0x02c57e1b to 127.0.0.1:62054 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-25 02:53:45,462 DEBUG [Listener at localhost/41915] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@dbb9ac1, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-25 02:53:45,463 DEBUG [Listener at localhost/41915] zookeeper.ReadOnlyZKClient(139): Connect 0x49d71120 to 127.0.0.1:62054 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-25 02:53:45,467 DEBUG [Listener at localhost/41915] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6fef7a21, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-25 02:53:45,467 DEBUG [Listener at localhost/41915] zookeeper.ReadOnlyZKClient(139): Connect 0x59d6b6ea to 127.0.0.1:62054 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-25 02:53:45,496 DEBUG [Listener at localhost/41915] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2879e2e1, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-25 02:53:45,496 DEBUG [Listener at localhost/41915] zookeeper.ReadOnlyZKClient(139): Connect 0x0d24bbb7 to 127.0.0.1:62054 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-25 02:53:45,505 DEBUG [Listener at localhost/41915] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6aad2e18, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-25 02:53:45,506 DEBUG [Listener at localhost/41915] zookeeper.ReadOnlyZKClient(139): Connect 0x63871a9d to 127.0.0.1:62054 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-25 02:53:45,510 DEBUG [Listener at localhost/41915] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4bd6fc59, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-25 02:53:45,525 DEBUG [hconnection-0xd2f522e-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-25 02:53:45,525 DEBUG [hconnection-0x5155e340-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-25 02:53:45,530 DEBUG [hconnection-0x6d07d495-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-25 02:53:45,532 DEBUG [hconnection-0x18ccc8f3-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-25 02:53:45,532 DEBUG [hconnection-0x3da37eba-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-25 02:53:45,533 DEBUG [hconnection-0x264fafb5-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-25 02:53:45,539 DEBUG [hconnection-0xd322817-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-25 02:53:45,542 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:43794, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-25 02:53:45,549 DEBUG [hconnection-0x1e073989-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-25 02:53:45,549 DEBUG [hconnection-0x3835db66-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-25 02:53:45,549 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:43804, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-25 02:53:45,550 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:43806, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-25 02:53:45,550 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:53:45,551 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:43828, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-25 02:53:45,558 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:43822, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-25 02:53:45,558 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:43854, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-25 02:53:45,560 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:43862, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-25 02:53:45,573 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:43840, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-25 02:53:45,574 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:43864, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-25 02:53:45,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:53:45,586 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:53:45,586 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:53:45,587 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:53:45,587 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:53:45,588 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:45,588 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:53:45,590 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:53:45,590 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:53:45,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:53:45,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:53:45,591 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:45,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:53:45,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:53:45,592 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:53:45,594 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:45,595 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:45,595 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:53:45,600 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:53:45,608 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:53:45,608 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:53:45,613 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:53:45,616 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:53:45,616 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:53:45,617 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. 2023-04-25 02:53:45,617 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. started... 2023-04-25 02:53:45,617 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 047cf722b8f6bff032073f994134764a 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-25 02:53:45,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(9158): Flush requested on 047cf722b8f6bff032073f994134764a 2023-04-25 02:53:45,619 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. as already flushing 2023-04-25 02:53:45,631 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 047cf722b8f6bff032073f994134764a, store=A 2023-04-25 02:53:45,632 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:53:45,633 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 047cf722b8f6bff032073f994134764a, store=B 2023-04-25 02:53:45,634 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:53:45,634 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 047cf722b8f6bff032073f994134764a, store=C 2023-04-25 02:53:45,634 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:53:45,800 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:53:45,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43828 deadline: 1682391285768, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:45,807 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:53:45,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43864 deadline: 1682391285771, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:45,810 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682391225759, max timestamp=1682391225785Num uniques 859; 2023-04-25 02:53:45,810 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-25 02:53:45,819 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1682391225759, max timestamp=1682391225773Num uniques 859; 2023-04-25 02:53:45,820 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-04-25 02:53:45,821 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1682391225759, max timestamp=1682391225773Num uniques 859; 2023-04-25 02:53:45,822 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-04-25 02:53:45,866 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:53:45,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43822 deadline: 1682391285856, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:45,866 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:53:45,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43840 deadline: 1682391285858, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:45,869 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:53:45,870 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43806 deadline: 1682391285867, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:45,917 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:53:45,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43828 deadline: 1682391285909, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:45,917 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:53:45,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43864 deadline: 1682391285910, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:45,973 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:53:45,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43822 deadline: 1682391285970, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:45,974 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:53:45,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43840 deadline: 1682391285971, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:45,975 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:53:45,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43806 deadline: 1682391285973, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:46,124 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:53:46,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43828 deadline: 1682391286123, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:46,126 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:53:46,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43864 deadline: 1682391286125, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:46,183 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:53:46,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43840 deadline: 1682391286179, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:46,184 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:53:46,184 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:53:46,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43822 deadline: 1682391286180, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:46,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43806 deadline: 1682391286180, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:46,233 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=12 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/.tmp/A/fb1f6146377f462f98c9d568f13113d1 2023-04-25 02:53:46,430 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=12 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/.tmp/B/c6eb3e4c7d404fddb2a1a2a946cb4489 2023-04-25 02:53:46,437 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:53:46,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43864 deadline: 1682391286431, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:46,439 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:53:46,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43828 deadline: 1682391286432, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:46,489 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:53:46,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43806 deadline: 1682391286488, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:46,492 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:53:46,493 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:53:46,493 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43822 deadline: 1682391286492, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:46,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43840 deadline: 1682391286492, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:46,902 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=12 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/.tmp/C/e1c283bc917340499cff80cee4e3ccef 2023-04-25 02:53:46,913 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/.tmp/A/fb1f6146377f462f98c9d568f13113d1 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/A/fb1f6146377f462f98c9d568f13113d1 2023-04-25 02:53:46,923 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/A/fb1f6146377f462f98c9d568f13113d1, entries=150, sequenceid=12, filesize=11.6 K 2023-04-25 02:53:46,930 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/.tmp/B/c6eb3e4c7d404fddb2a1a2a946cb4489 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/B/c6eb3e4c7d404fddb2a1a2a946cb4489 2023-04-25 02:53:46,945 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/B/c6eb3e4c7d404fddb2a1a2a946cb4489, entries=150, sequenceid=12, filesize=11.6 K 2023-04-25 02:53:46,948 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/.tmp/C/e1c283bc917340499cff80cee4e3ccef as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/C/e1c283bc917340499cff80cee4e3ccef 2023-04-25 02:53:46,972 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/C/e1c283bc917340499cff80cee4e3ccef, entries=150, sequenceid=12, filesize=11.6 K 2023-04-25 02:53:46,976 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=126.68 KB/129721 for 047cf722b8f6bff032073f994134764a in 1358ms, sequenceid=12, compaction requested=false 2023-04-25 02:53:46,978 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'TestAcidGuarantees' 2023-04-25 02:53:46,980 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 047cf722b8f6bff032073f994134764a: 2023-04-25 02:53:46,981 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. 2023-04-25 02:53:46,984 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:53:46,985 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:53:46,985 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:46,985 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:53:46,985 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:53:46,988 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:46,988 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:46,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:46,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:53:46,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:53:46,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:53:46,990 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:46,990 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:53:46,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:53:46,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:46,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:46,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:53:46,995 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:53:46,995 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@61a24458[Count = 0] remaining members to acquire global barrier 2023-04-25 02:53:46,995 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:53:46,995 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:47,003 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:47,003 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:47,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:47,003 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:53:47,003 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:53:47,003 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:53:47,005 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:47,005 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:53:47,009 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:47,009 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:53:47,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:47,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:53:47,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:53:47,009 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:53:47,010 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:53:47,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:53:47,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:53:47,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:47,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:47,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:53:47,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:47,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:47,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:53:47,021 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:53:47,021 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:53:47,021 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:53:47,021 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:53:47,021 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:53:47,021 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:53:47,025 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:47,025 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:53:47,025 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:47,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:47,026 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:47,026 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:53:47,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:53:47,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:47,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:53:47,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:53:47,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:47,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:53:47,030 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:47,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:47,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:53:47,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:47,032 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 047cf722b8f6bff032073f994134764a 3/3 column families, dataSize=133.39 KB heapSize=383.16 KB 2023-04-25 02:53:47,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(9158): Flush requested on 047cf722b8f6bff032073f994134764a 2023-04-25 02:53:47,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:47,033 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:47,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:53:47,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:47,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:47,052 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 047cf722b8f6bff032073f994134764a, store=A 2023-04-25 02:53:47,053 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-25 02:53:47,053 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 047cf722b8f6bff032073f994134764a, store=B 2023-04-25 02:53:47,053 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-25 02:53:47,053 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 047cf722b8f6bff032073f994134764a, store=C 2023-04-25 02:53:47,057 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-25 02:53:47,063 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:53:47,063 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:47,063 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:53:47,063 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:53:47,063 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:53:47,064 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:53:47,063 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:53:47,063 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:53:47,064 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:53:47,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:53:47,067 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:47,067 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:47,067 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:47,067 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:47,067 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:53:47,068 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:53:47,068 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:53:47,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:53:47,102 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:53:47,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43840 deadline: 1682391287072, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:47,103 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:53:47,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43822 deadline: 1682391287076, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:47,118 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:53:47,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43864 deadline: 1682391287091, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:47,120 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:53:47,121 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43828 deadline: 1682391287099, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:47,132 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:53:47,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43806 deadline: 1682391287103, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:47,159 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.57 KB at sequenceid=50 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/.tmp/A/8c319e53504e434e92f0783fec8e57a1 2023-04-25 02:53:47,168 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:53:47,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:53:47,181 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:53:47,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:53:47,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:53:47,213 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:53:47,213 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:53:47,215 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:53:47,215 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:53:47,217 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:47,217 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:53:47,220 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:53:47,220 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:53:47,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:53:47,220 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:47,221 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:53:47,221 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:47,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43840 deadline: 1682391287206, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:47,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:53:47,224 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:53:47,225 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:53:47,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43822 deadline: 1682391287206, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:47,225 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:47,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:53:47,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:53:47,226 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:53:47,230 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:53:47,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43864 deadline: 1682391287221, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:47,240 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:53:47,240 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:53:47,240 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:53:47,240 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:53:47,243 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:53:47,243 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43828 deadline: 1682391287236, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:47,254 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:53:47,258 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:53:47,264 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:53:47,264 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. 2023-04-25 02:53:47,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43806 deadline: 1682391287241, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:47,264 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:53:47,277 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. started... 2023-04-25 02:53:47,277 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. as already flushing 2023-04-25 02:53:47,277 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. 2023-04-25 02:53:47,286 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:53:47,286 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:53:47,286 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:47,286 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:53:47,286 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:53:47,291 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:47,291 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:47,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:47,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:53:47,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:53:47,292 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:47,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:53:47,292 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:53:47,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:53:47,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:47,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:47,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:53:47,295 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:53:47,296 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7f6dbfd1[Count = 0] remaining members to acquire global barrier 2023-04-25 02:53:47,296 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:53:47,296 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:47,296 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=50 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/.tmp/B/a480c509a8124c1e9216dcba81c7f1e3 2023-04-25 02:53:47,297 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:47,297 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:47,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:47,298 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:53:47,298 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:53:47,298 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:53:47,298 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:47,298 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:53:47,301 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:53:47,301 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:47,301 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:53:47,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:47,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:53:47,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:53:47,302 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:53:47,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:53:47,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:53:47,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:47,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:47,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:53:47,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:47,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:47,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:53:47,315 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:53:47,315 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:53:47,315 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:53:47,315 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:53:47,315 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:53:47,316 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:53:47,322 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:47,322 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:53:47,322 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:47,322 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:47,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:47,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:53:47,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:47,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:53:47,323 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:53:47,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:53:47,323 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:47,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:53:47,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:47,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:47,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:53:47,326 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:47,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:47,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:47,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:53:47,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:47,343 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:53:47,343 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:53:47,344 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:53:47,343 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:47,344 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:53:47,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:53:47,344 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:53:47,344 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:53:47,344 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:53:47,344 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:53:47,344 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:47,344 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:47,344 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:47,344 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:47,344 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:53:47,344 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:53:47,345 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:53:47,345 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:53:47,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:53:47,445 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:53:47,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:53:47,451 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:53:47,452 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:53:47,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43864 deadline: 1682391287433, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:47,455 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:53:47,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43828 deadline: 1682391287446, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:47,497 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.57 KB at sequenceid=50 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/.tmp/C/cc68fe1357cb40e083f89f1f51013e29 2023-04-25 02:53:47,508 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:53:47,508 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:53:47,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43806 deadline: 1682391287498, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:47,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43840 deadline: 1682391287501, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:47,514 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/.tmp/A/8c319e53504e434e92f0783fec8e57a1 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/A/8c319e53504e434e92f0783fec8e57a1 2023-04-25 02:53:47,519 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:53:47,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43822 deadline: 1682391287508, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:47,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:53:47,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:53:47,526 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:53:47,526 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:53:47,528 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:53:47,528 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:53:47,540 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:47,541 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:53:47,545 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:47,545 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:53:47,546 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:53:47,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:53:47,547 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/A/8c319e53504e434e92f0783fec8e57a1, entries=150, sequenceid=50, filesize=11.6 K 2023-04-25 02:53:47,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:53:47,559 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:47,559 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:47,560 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:53:47,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:53:47,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:53:47,560 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:53:47,562 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/.tmp/B/a480c509a8124c1e9216dcba81c7f1e3 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/B/a480c509a8124c1e9216dcba81c7f1e3 2023-04-25 02:53:47,572 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:53:47,576 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:53:47,576 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:53:47,576 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:53:47,574 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/B/a480c509a8124c1e9216dcba81c7f1e3, entries=150, sequenceid=50, filesize=11.6 K 2023-04-25 02:53:47,580 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:53:47,586 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/.tmp/C/cc68fe1357cb40e083f89f1f51013e29 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/C/cc68fe1357cb40e083f89f1f51013e29 2023-04-25 02:53:47,587 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:53:47,587 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:53:47,587 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. 2023-04-25 02:53:47,588 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. started... 2023-04-25 02:53:47,588 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. as already flushing 2023-04-25 02:53:47,588 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. 2023-04-25 02:53:47,593 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:53:47,593 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:53:47,593 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:47,593 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:53:47,593 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:53:47,598 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:47,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:47,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:53:47,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:53:47,598 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:47,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:53:47,599 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:47,599 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:53:47,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:53:47,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:47,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:47,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:53:47,603 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:53:47,603 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3615f4b7[Count = 0] remaining members to acquire global barrier 2023-04-25 02:53:47,603 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:53:47,603 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:47,606 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:47,606 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:47,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:47,607 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:53:47,607 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:53:47,607 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:53:47,607 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:47,608 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:53:47,612 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:47,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:47,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:53:47,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:53:47,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:53:47,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:53:47,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:47,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:47,615 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:53:47,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:53:47,615 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:53:47,616 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:53:47,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:47,623 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/C/cc68fe1357cb40e083f89f1f51013e29, entries=150, sequenceid=50, filesize=11.6 K 2023-04-25 02:53:47,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:47,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:53:47,627 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:53:47,627 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:53:47,627 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:53:47,627 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:53:47,627 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:53:47,627 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:53:47,627 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~140.10 KB/143461, heapSize ~403.73 KB/413424, currentSize=40.25 KB/41220 for 047cf722b8f6bff032073f994134764a in 595ms, sequenceid=50, compaction requested=false 2023-04-25 02:53:47,633 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 047cf722b8f6bff032073f994134764a: 2023-04-25 02:53:47,638 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:47,638 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:47,639 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:47,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:47,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:47,638 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:53:47,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:53:47,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:53:47,639 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:53:47,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:53:47,639 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:47,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:53:47,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:47,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:47,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:53:47,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:47,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:47,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:53:47,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:47,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:47,649 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:47,664 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:53:47,664 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:53:47,664 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:47,664 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:53:47,664 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:53:47,664 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:53:47,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:53:47,665 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:53:47,665 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:53:47,665 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:53:47,665 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:53:47,665 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:53:47,666 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:53:47,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:53:47,664 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:47,668 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:47,668 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:47,668 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:47,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(9158): Flush requested on 047cf722b8f6bff032073f994134764a 2023-04-25 02:53:47,759 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 047cf722b8f6bff032073f994134764a 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-25 02:53:47,759 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 047cf722b8f6bff032073f994134764a, store=A 2023-04-25 02:53:47,759 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:53:47,760 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 047cf722b8f6bff032073f994134764a, store=B 2023-04-25 02:53:47,760 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:53:47,760 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 047cf722b8f6bff032073f994134764a, store=C 2023-04-25 02:53:47,760 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:53:47,766 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:53:47,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:53:47,771 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:53:47,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:53:47,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:53:47,786 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:53:47,786 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:53:47,788 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:53:47,788 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:53:47,789 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:47,789 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:53:47,791 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:53:47,791 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:47,791 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:53:47,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:53:47,791 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:47,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:53:47,791 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:53:47,792 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:47,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:53:47,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:53:47,793 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:53:47,806 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:53:47,807 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:53:47,807 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:53:47,812 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:53:47,816 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:53:47,817 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:53:47,817 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:53:47,817 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. 2023-04-25 02:53:47,817 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. started... 2023-04-25 02:53:47,817 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. as already flushing 2023-04-25 02:53:47,817 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. 2023-04-25 02:53:47,818 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:53:47,818 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:53:47,818 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:47,818 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:53:47,818 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:53:47,846 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:47,861 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:47,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:47,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:53:47,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:53:47,864 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:47,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:53:47,864 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:53:47,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:53:47,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:47,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:47,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:53:47,866 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:53:47,867 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@19e681a9[Count = 0] remaining members to acquire global barrier 2023-04-25 02:53:47,867 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:53:47,867 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:47,868 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:47,868 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:47,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:47,869 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:53:47,869 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:53:47,869 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:53:47,869 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:47,869 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:53:47,874 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:53:47,874 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:47,874 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:53:47,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:47,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:53:47,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:53:47,874 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:53:47,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:53:47,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:53:47,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:47,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:47,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:53:47,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:47,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:47,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:53:47,884 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:53:47,884 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:53:47,884 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:53:47,884 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:53:47,884 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:53:47,884 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:53:47,887 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:47,887 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:53:47,887 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:47,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:47,887 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:47,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:47,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:53:47,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:53:47,888 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:53:47,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:53:47,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:53:47,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:47,888 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:47,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:47,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:53:47,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:47,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:47,891 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:47,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:53:47,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:47,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:47,899 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:47,899 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:53:47,899 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:53:47,899 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:53:47,899 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:53:47,900 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:53:47,900 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:53:47,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:53:47,899 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:53:47,899 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:47,900 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:53:47,900 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:53:47,901 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:53:47,900 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:47,901 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:53:47,901 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:47,901 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:47,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:53:47,901 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:53:47,951 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:53:47,951 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1682391227855, max timestamp=1682391227907Num uniques 859; 2023-04-25 02:53:47,951 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43828 deadline: 1682391287908, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:47,952 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-04-25 02:53:47,958 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682391227855, max timestamp=1682391227907Num uniques 859; 2023-04-25 02:53:47,958 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-25 02:53:48,001 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:53:48,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:53:48,007 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:53:48,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:53:48,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:53:48,011 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:53:48,012 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:53:48,013 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:53:48,013 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:53:48,014 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:48,014 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:53:48,016 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:53:48,016 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:53:48,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:53:48,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:53:48,017 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:48,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:53:48,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:53:48,018 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:53:48,020 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:48,020 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:48,020 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:53:48,027 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:53:48,027 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:53:48,027 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:53:48,027 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:53:48,030 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:53:48,036 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:53:48,036 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:53:48,036 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. 2023-04-25 02:53:48,036 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. started... 2023-04-25 02:53:48,036 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. as already flushing 2023-04-25 02:53:48,040 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. 2023-04-25 02:53:48,043 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:53:48,043 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:53:48,043 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:48,043 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:53:48,043 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:53:48,047 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:48,047 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:48,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:48,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:53:48,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:53:48,048 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:48,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:53:48,048 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:53:48,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:53:48,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:48,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:48,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:53:48,050 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:53:48,050 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6a158649[Count = 0] remaining members to acquire global barrier 2023-04-25 02:53:48,050 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:53:48,050 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:48,052 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:48,052 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:48,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:48,053 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:48,053 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:53:48,054 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:53:48,054 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:53:48,054 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:53:48,057 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:53:48,057 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:48,057 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:53:48,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:48,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:53:48,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:53:48,057 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:53:48,065 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-25 02:53:48,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:53:48,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:53:48,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:48,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:48,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:53:48,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:48,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:48,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:53:48,069 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:53:48,069 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:53:48,069 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:53:48,069 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:53:48,070 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:53:48,070 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:53:48,072 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:48,072 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:48,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:48,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:53:48,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:53:48,072 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:48,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:48,072 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:53:48,073 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:53:48,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:53:48,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:53:48,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:48,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:48,074 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:48,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:53:48,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:48,081 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:53:48,081 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43822 deadline: 1682391288057, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:48,081 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:53:48,081 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43864 deadline: 1682391288054, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:48,083 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:48,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:48,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:53:48,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:48,084 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:53:48,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43806 deadline: 1682391288069, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:48,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:48,090 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:53:48,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43840 deadline: 1682391288078, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:48,106 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:48,106 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:53:48,106 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:53:48,106 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:48,106 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:48,106 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:48,106 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:48,107 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:53:48,107 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:53:48,107 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:53:48,105 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:53:48,108 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:53:48,108 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:53:48,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:53:48,108 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:53:48,108 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:53:48,108 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:53:48,108 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:53:48,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:53:48,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43828 deadline: 1682391288085, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:48,199 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:53:48,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43822 deadline: 1682391288184, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:48,201 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:53:48,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43864 deadline: 1682391288195, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:48,201 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:53:48,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43806 deadline: 1682391288196, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:48,208 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:53:48,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:53:48,212 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:53:48,223 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:53:48,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43828 deadline: 1682391288213, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:48,224 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:53:48,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43840 deadline: 1682391288214, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:48,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:53:48,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:53:48,230 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:53:48,230 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:53:48,231 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:53:48,231 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:53:48,232 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:meta' 2023-04-25 02:53:48,232 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:48,232 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:53:48,234 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:53:48,234 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:53:48,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:53:48,234 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:48,234 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:namespace' 2023-04-25 02:53:48,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:53:48,235 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:48,235 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:53:48,235 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:48,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:53:48,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:53:48,236 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:53:48,236 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:53:48,236 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:53:48,236 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:53:48,236 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:53:48,238 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:53:48,248 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:53:48,248 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. 2023-04-25 02:53:48,248 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:53:48,250 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=60 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/.tmp/A/30024e1bd1f44345b0cf6d4b6d9c89b7 2023-04-25 02:53:48,248 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. started... 2023-04-25 02:53:48,255 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. as already flushing 2023-04-25 02:53:48,258 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. 2023-04-25 02:53:48,275 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:53:48,275 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:53:48,276 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:48,276 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:53:48,276 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:53:48,281 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:48,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:48,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:53:48,281 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:48,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:53:48,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:53:48,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:53:48,283 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:48,283 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:53:48,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:48,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:48,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:53:48,290 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:53:48,290 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@78142fed[Count = 0] remaining members to acquire global barrier 2023-04-25 02:53:48,290 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:53:48,290 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:48,296 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:48,296 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:48,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:48,299 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:48,299 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:53:48,300 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:53:48,300 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:53:48,300 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:53:48,303 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:48,303 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:53:48,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:48,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:53:48,303 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:53:48,303 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:53:48,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:53:48,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:53:48,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:53:48,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:48,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:48,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:53:48,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:48,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:48,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:53:48,308 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:53:48,308 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:53:48,309 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:53:48,309 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:53:48,309 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:53:48,309 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:53:48,311 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:48,311 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:53:48,311 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:48,311 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:48,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:48,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:48,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:53:48,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:53:48,313 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:53:48,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:53:48,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:53:48,313 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:48,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:48,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:48,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:53:48,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:48,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:48,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:53:48,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:48,317 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:48,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:48,330 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:48,330 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:53:48,330 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:53:48,330 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:53:48,331 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:53:48,331 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:53:48,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:53:48,331 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:48,331 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:48,331 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:48,331 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:48,331 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:53:48,331 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:53:48,331 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:53:48,332 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:53:48,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:53:48,332 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:53:48,332 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:53:48,404 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:53:48,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43822 deadline: 1682391288403, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:48,405 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:53:48,405 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:53:48,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43864 deadline: 1682391288405, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:48,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43806 deadline: 1682391288404, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:48,431 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:53:48,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43828 deadline: 1682391288427, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:48,432 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:53:48,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43840 deadline: 1682391288427, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:48,432 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:53:48,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:53:48,436 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:53:48,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:53:48,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:53:48,445 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:53:48,445 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:53:48,446 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:53:48,446 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:53:48,447 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:48,447 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:53:48,452 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:53:48,452 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:53:48,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:53:48,453 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:48,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:53:48,453 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:48,453 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:53:48,454 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:48,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:53:48,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:53:48,454 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:53:48,455 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:53:48,455 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:53:48,456 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:53:48,456 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:53:48,457 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:53:48,460 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:53:48,460 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:53:48,460 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. 2023-04-25 02:53:48,460 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. started... 2023-04-25 02:53:48,460 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. as already flushing 2023-04-25 02:53:48,460 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. 2023-04-25 02:53:48,466 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:53:48,466 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:53:48,466 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:48,466 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:53:48,466 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:53:48,468 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:48,468 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:48,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:48,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:53:48,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:53:48,469 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:48,469 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:53:48,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:53:48,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:53:48,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:48,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:48,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:53:48,473 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:53:48,473 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@61bdcc14[Count = 0] remaining members to acquire global barrier 2023-04-25 02:53:48,473 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:53:48,473 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:48,475 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:48,475 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:48,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:48,475 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:53:48,475 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:53:48,475 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:48,475 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:53:48,475 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:53:48,477 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:53:48,477 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:53:48,477 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:48,477 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:53:48,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:48,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:53:48,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:53:48,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:53:48,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:53:48,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:48,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:48,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:53:48,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:48,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:48,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:53:48,486 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:53:48,486 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:53:48,486 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:53:48,486 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:53:48,486 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:53:48,486 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:53:48,490 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:48,490 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:53:48,490 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:48,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:48,491 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:53:48,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:53:48,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:48,492 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:48,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:48,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:53:48,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:53:48,497 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:48,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:53:48,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:48,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:53:48,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:48,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:48,500 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:48,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:53:48,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:48,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:48,504 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:53:48,504 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:53:48,504 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:53:48,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:53:48,504 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:48,504 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:53:48,505 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:53:48,505 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:48,505 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:48,505 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:48,505 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:48,505 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:53:48,505 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:53:48,505 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:53:48,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:53:48,505 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:53:48,509 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:53:48,509 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:53:48,509 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:53:48,609 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:53:48,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:53:48,616 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:53:48,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:53:48,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:53:48,637 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:53:48,637 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:53:48,638 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:53:48,639 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:53:48,639 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:48,640 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:53:48,643 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:48,643 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:53:48,643 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:53:48,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:53:48,645 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:48,645 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:53:48,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:53:48,646 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:48,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:53:48,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:53:48,646 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:53:48,660 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:53:48,660 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:53:48,660 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:53:48,660 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:53:48,665 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:53:48,668 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:53:48,668 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. 2023-04-25 02:53:48,668 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:53:48,668 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. started... 2023-04-25 02:53:48,669 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. as already flushing 2023-04-25 02:53:48,669 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. 2023-04-25 02:53:48,669 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:53:48,669 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:53:48,669 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:48,669 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:53:48,669 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:53:48,672 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:48,672 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:48,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:48,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:53:48,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:53:48,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:53:48,673 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:48,673 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:53:48,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:53:48,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:48,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:48,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:53:48,675 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:53:48,675 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:53:48,675 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:48,675 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4c7bf243[Count = 0] remaining members to acquire global barrier 2023-04-25 02:53:48,678 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:48,678 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:48,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:48,679 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:53:48,679 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:53:48,679 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:53:48,679 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:48,679 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:53:48,681 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:53:48,681 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:53:48,681 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:53:48,686 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:48,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:48,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:53:48,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:53:48,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:53:48,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:53:48,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:48,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:48,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:53:48,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:48,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:48,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:53:48,691 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:53:48,691 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:53:48,691 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:53:48,691 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:53:48,691 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:53:48,691 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:53:48,694 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:48,694 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:48,694 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:48,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:48,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:53:48,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:53:48,694 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:53:48,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:48,695 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:53:48,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:53:48,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:48,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:53:48,696 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:48,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:48,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:53:48,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:48,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:48,699 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:48,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:53:48,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:48,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:48,707 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:53:48,707 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:53:48,707 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:53:48,707 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:48,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:53:48,708 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:53:48,707 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:53:48,708 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:53:48,708 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:53:48,709 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:53:48,709 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:48,709 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:48,709 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:53:48,710 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:53:48,709 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:48,710 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:48,712 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:53:48,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43864 deadline: 1682391288708, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:48,713 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:53:48,713 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:53:48,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43822 deadline: 1682391288710, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:48,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:53:48,714 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:53:48,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43806 deadline: 1682391288710, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:48,735 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:53:48,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43840 deadline: 1682391288735, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:48,738 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:53:48,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43828 deadline: 1682391288737, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:48,769 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=60 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/.tmp/B/cbc5f7506be1404695aff07919df8ad1 2023-04-25 02:53:48,810 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:53:48,811 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:53:48,815 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:53:48,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:53:48,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:53:48,833 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:53:48,833 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:53:48,853 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:53:48,853 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:53:48,854 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:48,854 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:53:48,867 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:53:48,870 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=60 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/.tmp/C/91aef605175a4841876534a48a41031a 2023-04-25 02:53:48,875 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:53:48,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:53:48,876 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:48,883 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/.tmp/A/30024e1bd1f44345b0cf6d4b6d9c89b7 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/A/30024e1bd1f44345b0cf6d4b6d9c89b7 2023-04-25 02:53:48,885 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:48,885 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:53:48,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:53:48,885 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:48,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:53:48,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:53:48,886 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:53:48,890 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:53:48,890 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:53:48,890 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:53:48,890 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:53:48,903 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:53:48,904 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:53:48,904 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. 2023-04-25 02:53:48,905 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. started... 2023-04-25 02:53:48,904 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:53:48,905 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. as already flushing 2023-04-25 02:53:48,905 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. 2023-04-25 02:53:48,905 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:53:48,905 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:53:48,905 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:48,905 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:53:48,905 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:53:48,907 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:48,908 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:48,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:48,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:53:48,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:53:48,908 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:48,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:53:48,908 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:53:48,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:53:48,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:48,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:48,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:53:48,910 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:53:48,910 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@a9a024b[Count = 0] remaining members to acquire global barrier 2023-04-25 02:53:48,910 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:53:48,910 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:48,913 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:48,913 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:48,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:48,913 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:53:48,913 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:53:48,913 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:53:48,914 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:48,914 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:53:48,916 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:53:48,916 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:53:48,916 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:53:48,918 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:48,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:48,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:53:48,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:53:48,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:53:48,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:53:48,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:48,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:48,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:53:48,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:48,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:48,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:53:48,929 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:53:48,929 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:53:48,929 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:53:48,929 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:53:48,929 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:53:48,929 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:53:48,931 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/A/30024e1bd1f44345b0cf6d4b6d9c89b7, entries=150, sequenceid=60, filesize=11.6 K 2023-04-25 02:53:48,932 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:48,932 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:53:48,932 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:48,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:48,932 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:48,933 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:53:48,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:53:48,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:48,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:48,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:53:48,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:53:48,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:53:48,935 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:48,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:48,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:53:48,936 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/.tmp/B/cbc5f7506be1404695aff07919df8ad1 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/B/cbc5f7506be1404695aff07919df8ad1 2023-04-25 02:53:48,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:48,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:48,940 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:48,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:53:48,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:48,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:48,946 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:48,946 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:53:48,946 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:53:48,946 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:53:48,946 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:53:48,946 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:53:48,946 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:53:48,946 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:48,946 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:53:48,946 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:48,946 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:48,947 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:48,946 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:53:48,947 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:53:48,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:53:48,947 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:53:48,947 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:53:48,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:53:48,971 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/B/cbc5f7506be1404695aff07919df8ad1, entries=100, sequenceid=60, filesize=9.3 K 2023-04-25 02:53:48,977 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/.tmp/C/91aef605175a4841876534a48a41031a as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/C/91aef605175a4841876534a48a41031a 2023-04-25 02:53:48,991 DEBUG [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(130): Registering adapter for the MetricRegistry: RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-04-25 02:53:48,993 INFO [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(134): Registering RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint Metrics about HBase RegionObservers 2023-04-25 02:53:49,003 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/C/91aef605175a4841876534a48a41031a, entries=100, sequenceid=60, filesize=9.3 K 2023-04-25 02:53:49,009 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=138.26 KB/141575 for 047cf722b8f6bff032073f994134764a in 1250ms, sequenceid=60, compaction requested=true 2023-04-25 02:53:49,010 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 047cf722b8f6bff032073f994134764a: 2023-04-25 02:53:49,013 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 02:53:49,013 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 02:53:49,013 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 02:53:49,014 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-25 02:53:49,016 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-25 02:53:49,026 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 33314 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-25 02:53:49,029 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HStore(1912): 047cf722b8f6bff032073f994134764a/B is initiating minor compaction (all files) 2023-04-25 02:53:49,029 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 047cf722b8f6bff032073f994134764a/B in TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. 2023-04-25 02:53:49,030 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/B/c6eb3e4c7d404fddb2a1a2a946cb4489, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/B/a480c509a8124c1e9216dcba81c7f1e3, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/B/cbc5f7506be1404695aff07919df8ad1] into tmpdir=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/.tmp, totalSize=32.5 K 2023-04-25 02:53:49,032 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting c6eb3e4c7d404fddb2a1a2a946cb4489, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=12, earliestPutTs=1682391225600 2023-04-25 02:53:49,032 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35654 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-25 02:53:49,037 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HStore(1912): 047cf722b8f6bff032073f994134764a/A is initiating minor compaction (all files) 2023-04-25 02:53:49,037 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 047cf722b8f6bff032073f994134764a/A in TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. 2023-04-25 02:53:49,037 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/A/fb1f6146377f462f98c9d568f13113d1, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/A/8c319e53504e434e92f0783fec8e57a1, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/A/30024e1bd1f44345b0cf6d4b6d9c89b7] into tmpdir=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/.tmp, totalSize=34.8 K 2023-04-25 02:53:49,038 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting a480c509a8124c1e9216dcba81c7f1e3, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1682391225855 2023-04-25 02:53:49,039 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting fb1f6146377f462f98c9d568f13113d1, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=12, earliestPutTs=1682391225600 2023-04-25 02:53:49,041 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting cbc5f7506be1404695aff07919df8ad1, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=60, earliestPutTs=1682391227085 2023-04-25 02:53:49,042 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting 8c319e53504e434e92f0783fec8e57a1, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1682391225855 2023-04-25 02:53:49,045 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting 30024e1bd1f44345b0cf6d4b6d9c89b7, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=60, earliestPutTs=1682391227085 2023-04-25 02:53:49,047 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:53:49,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:53:49,066 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:53:49,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:53:49,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:53:49,084 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:53:49,084 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:53:49,085 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:53:49,085 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:53:49,086 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:49,086 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:53:49,092 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] throttle.PressureAwareThroughputController(145): 047cf722b8f6bff032073f994134764a#A#compaction#9 average throughput is 1.31 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-25 02:53:49,096 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:49,096 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:53:49,096 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:53:49,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:53:49,096 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:49,096 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:53:49,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:53:49,097 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:49,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:53:49,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:53:49,098 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:53:49,104 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:53:49,104 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:53:49,104 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:53:49,104 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:53:49,107 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:53:49,108 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:53:49,108 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:53:49,108 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. 2023-04-25 02:53:49,108 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. started... 2023-04-25 02:53:49,108 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 047cf722b8f6bff032073f994134764a 3/3 column families, dataSize=138.26 KB heapSize=402.32 KB 2023-04-25 02:53:49,110 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 047cf722b8f6bff032073f994134764a, store=A 2023-04-25 02:53:49,110 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 02:53:49,110 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 047cf722b8f6bff032073f994134764a, store=B 2023-04-25 02:53:49,110 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 02:53:49,110 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 047cf722b8f6bff032073f994134764a, store=C 2023-04-25 02:53:49,110 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 02:53:49,113 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] throttle.PressureAwareThroughputController(145): 047cf722b8f6bff032073f994134764a#B#compaction#10 average throughput is 0.73 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-25 02:53:49,188 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/.tmp/A/cc671b2ab2ae4be78ebbe8537bc3c01b as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/A/cc671b2ab2ae4be78ebbe8537bc3c01b 2023-04-25 02:53:49,197 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.39 KB at sequenceid=93 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/.tmp/A/d0ee89d7f1d143a5a70d112d5ae6fb22 2023-04-25 02:53:49,215 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 047cf722b8f6bff032073f994134764a/A of 047cf722b8f6bff032073f994134764a into cc671b2ab2ae4be78ebbe8537bc3c01b(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 02:53:49,215 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 047cf722b8f6bff032073f994134764a: 2023-04-25 02:53:49,216 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a., storeName=047cf722b8f6bff032073f994134764a/A, priority=13, startTime=1682391229012; duration=0sec 2023-04-25 02:53:49,216 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 02:53:49,216 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-25 02:53:49,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(9158): Flush requested on 047cf722b8f6bff032073f994134764a 2023-04-25 02:53:49,224 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. as already flushing 2023-04-25 02:53:49,233 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:53:49,234 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43864 deadline: 1682391289233, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:49,238 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:53:49,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43806 deadline: 1682391289234, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:49,238 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:53:49,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43822 deadline: 1682391289234, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:49,245 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 33314 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-25 02:53:49,245 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HStore(1912): 047cf722b8f6bff032073f994134764a/C is initiating minor compaction (all files) 2023-04-25 02:53:49,245 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 047cf722b8f6bff032073f994134764a/C in TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. 2023-04-25 02:53:49,246 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/C/e1c283bc917340499cff80cee4e3ccef, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/C/cc68fe1357cb40e083f89f1f51013e29, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/C/91aef605175a4841876534a48a41031a] into tmpdir=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/.tmp, totalSize=32.5 K 2023-04-25 02:53:49,250 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting e1c283bc917340499cff80cee4e3ccef, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=12, earliestPutTs=1682391225600 2023-04-25 02:53:49,250 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:53:49,251 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:53:49,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43840 deadline: 1682391289239, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:49,252 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting cc68fe1357cb40e083f89f1f51013e29, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1682391225855 2023-04-25 02:53:49,251 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=93 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/.tmp/B/df13aedd944e410185ba3fadc0fb7e83 2023-04-25 02:53:49,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43828 deadline: 1682391289245, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:49,254 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting 91aef605175a4841876534a48a41031a, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=60, earliestPutTs=1682391227085 2023-04-25 02:53:49,311 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] throttle.PressureAwareThroughputController(145): 047cf722b8f6bff032073f994134764a#C#compaction#14 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-25 02:53:49,338 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:53:49,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43864 deadline: 1682391289336, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:49,362 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:53:49,362 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43806 deadline: 1682391289340, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:49,363 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:53:49,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43822 deadline: 1682391289345, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:49,390 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/.tmp/C/d978d53d0452462cb593a4a9d336b81a as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/C/d978d53d0452462cb593a4a9d336b81a 2023-04-25 02:53:49,402 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 047cf722b8f6bff032073f994134764a/C of 047cf722b8f6bff032073f994134764a into d978d53d0452462cb593a4a9d336b81a(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 02:53:49,402 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 047cf722b8f6bff032073f994134764a: 2023-04-25 02:53:49,402 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a., storeName=047cf722b8f6bff032073f994134764a/C, priority=13, startTime=1682391229013; duration=0sec 2023-04-25 02:53:49,403 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 02:53:49,541 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:53:49,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43864 deadline: 1682391289540, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:49,566 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:53:49,566 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:53:49,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43806 deadline: 1682391289565, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:49,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43822 deadline: 1682391289566, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:49,583 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/.tmp/B/80c60c631b7c454681537f934d1dd2b1 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/B/80c60c631b7c454681537f934d1dd2b1 2023-04-25 02:53:49,601 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 047cf722b8f6bff032073f994134764a/B of 047cf722b8f6bff032073f994134764a into 80c60c631b7c454681537f934d1dd2b1(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 02:53:49,601 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 047cf722b8f6bff032073f994134764a: 2023-04-25 02:53:49,601 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a., storeName=047cf722b8f6bff032073f994134764a/B, priority=13, startTime=1682391229013; duration=0sec 2023-04-25 02:53:49,601 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 02:53:49,732 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=67.09 KB at sequenceid=93 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/.tmp/C/79721a49470e41aab987a4934a3f715d 2023-04-25 02:53:49,744 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/.tmp/A/d0ee89d7f1d143a5a70d112d5ae6fb22 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/A/d0ee89d7f1d143a5a70d112d5ae6fb22 2023-04-25 02:53:49,753 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/A/d0ee89d7f1d143a5a70d112d5ae6fb22, entries=150, sequenceid=93, filesize=11.6 K 2023-04-25 02:53:49,756 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/.tmp/B/df13aedd944e410185ba3fadc0fb7e83 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/B/df13aedd944e410185ba3fadc0fb7e83 2023-04-25 02:53:49,767 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/B/df13aedd944e410185ba3fadc0fb7e83, entries=150, sequenceid=93, filesize=11.6 K 2023-04-25 02:53:49,772 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/.tmp/C/79721a49470e41aab987a4934a3f715d as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/C/79721a49470e41aab987a4934a3f715d 2023-04-25 02:53:49,793 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/C/79721a49470e41aab987a4934a3f715d, entries=150, sequenceid=93, filesize=11.6 K 2023-04-25 02:53:49,796 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~138.26 KB/141575, heapSize ~402.27 KB/411928, currentSize=40.25 KB/41220 for 047cf722b8f6bff032073f994134764a in 688ms, sequenceid=93, compaction requested=false 2023-04-25 02:53:49,797 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 047cf722b8f6bff032073f994134764a: 2023-04-25 02:53:49,797 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. 2023-04-25 02:53:49,797 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:53:49,799 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:53:49,799 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:49,799 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:53:49,799 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:53:49,802 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:49,802 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:49,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:49,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:53:49,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:53:49,803 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:49,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:53:49,803 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:53:49,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:53:49,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:49,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:49,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:53:49,805 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:53:49,805 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@53a82fb2[Count = 0] remaining members to acquire global barrier 2023-04-25 02:53:49,805 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:53:49,805 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:49,806 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:49,806 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:49,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:49,806 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:53:49,806 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:53:49,806 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:53:49,807 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:49,807 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:53:49,808 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:53:49,808 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:49,808 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:53:49,808 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:53:49,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:49,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:53:49,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:53:49,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:53:49,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:53:49,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:49,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:49,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:53:49,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:49,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:49,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:53:49,811 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:53:49,811 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:53:49,812 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:53:49,812 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:53:49,812 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:53:49,812 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:53:49,816 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:49,816 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:53:49,816 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:49,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:49,816 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:53:49,816 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:49,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:53:49,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:49,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:49,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:53:49,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:53:49,825 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:49,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:53:49,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:49,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:53:49,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:49,827 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:49,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:49,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:53:49,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:49,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:49,832 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:49,832 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:53:49,832 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:53:49,832 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:53:49,833 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:53:49,833 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:53:49,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:53:49,833 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:53:49,833 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:53:49,833 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:49,833 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:53:49,832 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:53:49,833 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:49,833 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:49,833 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:53:49,833 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:53:49,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:53:49,833 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:53:49,833 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:49,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(9158): Flush requested on 047cf722b8f6bff032073f994134764a 2023-04-25 02:53:49,850 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 047cf722b8f6bff032073f994134764a 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-25 02:53:49,850 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 047cf722b8f6bff032073f994134764a, store=A 2023-04-25 02:53:49,850 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:53:49,850 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 047cf722b8f6bff032073f994134764a, store=B 2023-04-25 02:53:49,851 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:53:49,851 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 047cf722b8f6bff032073f994134764a, store=C 2023-04-25 02:53:49,851 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:53:49,881 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=106 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/.tmp/A/9db33aa5cd3840b2ae0a076d53bc8f0d 2023-04-25 02:53:49,928 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:53:49,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43822 deadline: 1682391289924, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:49,929 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682391229899, max timestamp=1682391229924Num uniques 859; 2023-04-25 02:53:49,929 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-25 02:53:49,929 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682391229899, max timestamp=1682391229924Num uniques 859; 2023-04-25 02:53:49,929 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682391229899, max timestamp=1682391229921Num uniques 859; 2023-04-25 02:53:49,929 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-25 02:53:49,929 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-25 02:53:49,933 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:53:49,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:53:49,938 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:53:49,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:53:49,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:53:49,940 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:53:49,940 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:53:49,941 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:53:49,941 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:53:49,941 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:49,941 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:53:49,943 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:53:49,943 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:49,943 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:53:49,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:53:49,943 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:49,943 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:53:49,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:53:49,944 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:49,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:53:49,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:53:49,944 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:53:49,945 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:53:49,945 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:53:49,945 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:53:49,945 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:53:49,946 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:53:49,946 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:53:49,946 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:53:49,946 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. 2023-04-25 02:53:49,946 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. started... 2023-04-25 02:53:49,946 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. as already flushing 2023-04-25 02:53:49,946 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. 2023-04-25 02:53:49,946 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:53:49,946 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:53:49,946 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:49,947 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:53:49,947 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:53:49,948 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:49,948 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:49,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:49,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:53:49,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:53:49,949 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:49,949 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:53:49,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:53:49,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:53:49,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:49,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:49,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:53:49,952 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:53:49,952 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5c7a94bf[Count = 0] remaining members to acquire global barrier 2023-04-25 02:53:49,952 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:53:49,952 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:49,954 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:49,954 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:49,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:49,954 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:53:49,954 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:53:49,954 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:53:49,954 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:49,954 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:53:49,957 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:49,957 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:53:49,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:49,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:53:49,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:53:49,957 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:53:49,957 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:53:49,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:53:49,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:53:49,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:49,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:49,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:53:49,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:49,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:49,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:53:49,963 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:53:49,963 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:53:49,963 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:53:49,963 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:53:49,963 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:53:49,963 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:53:49,965 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:49,965 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:49,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:49,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:53:49,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:53:49,965 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:49,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:49,965 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:53:49,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:53:49,966 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:53:49,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:53:49,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:49,966 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:49,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:49,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:53:49,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:49,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:49,968 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:49,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:53:49,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:49,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:49,972 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:49,972 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:53:49,973 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:53:49,973 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:53:49,973 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:53:49,973 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:53:49,973 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:49,973 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:49,973 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:53:49,973 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:53:49,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:53:49,973 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:53:49,976 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:53:49,973 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:49,976 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:49,976 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:53:49,976 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:53:49,977 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:53:49,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:53:50,004 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:53:50,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43806 deadline: 1682391290001, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:50,006 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:53:50,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43864 deadline: 1682391290002, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:50,031 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:53:50,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43822 deadline: 1682391290030, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:50,076 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:53:50,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:53:50,081 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:53:50,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:53:50,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:53:50,092 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:53:50,092 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:53:50,098 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:53:50,098 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:53:50,098 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:50,099 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:53:50,100 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:53:50,101 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:53:50,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:53:50,101 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:50,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:53:50,102 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:50,102 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:53:50,103 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:50,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:53:50,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:53:50,104 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:53:50,109 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:53:50,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43806 deadline: 1682391290107, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:50,112 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:53:50,112 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:53:50,112 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:53:50,112 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:53:50,117 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:53:50,117 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:53:50,117 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:53:50,117 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. 2023-04-25 02:53:50,117 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:53:50,117 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. started... 2023-04-25 02:53:50,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43864 deadline: 1682391290110, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:50,118 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. as already flushing 2023-04-25 02:53:50,118 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. 2023-04-25 02:53:50,118 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:53:50,118 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:53:50,118 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:50,118 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:53:50,118 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:53:50,125 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:50,125 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:50,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:50,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:53:50,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:53:50,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:53:50,126 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:50,126 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:53:50,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:53:50,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:50,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:50,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:53:50,129 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:53:50,129 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4f809c99[Count = 0] remaining members to acquire global barrier 2023-04-25 02:53:50,129 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:53:50,129 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:50,130 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:50,130 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:50,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:50,130 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:53:50,131 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:53:50,131 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:53:50,131 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:50,131 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:53:50,132 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:53:50,132 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:50,132 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:53:50,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:50,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:53:50,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:53:50,132 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:53:50,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:53:50,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:53:50,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:50,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:50,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:53:50,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:50,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:50,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:53:50,144 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:53:50,144 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:53:50,144 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:53:50,144 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:53:50,144 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:53:50,144 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:53:50,149 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:50,149 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:50,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:50,150 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:50,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:53:50,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:53:50,149 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:53:50,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:50,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:53:50,150 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:53:50,150 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:50,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:50,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:53:50,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:50,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:53:50,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:50,152 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:50,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:50,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:53:50,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:50,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:50,156 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:50,156 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:53:50,156 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:53:50,156 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:50,156 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:50,156 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:50,156 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:50,156 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:53:50,157 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:53:50,157 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:53:50,157 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:53:50,164 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:53:50,164 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:53:50,165 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:53:50,165 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:53:50,165 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:53:50,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:53:50,165 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:53:50,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:53:50,238 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:53:50,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43822 deadline: 1682391290234, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:50,262 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:53:50,262 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43840 deadline: 1682391290259, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:50,264 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:53:50,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:53:50,266 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:53:50,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43828 deadline: 1682391290266, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:50,268 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:53:50,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:53:50,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:53:50,273 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:53:50,273 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:53:50,277 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:53:50,277 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:53:50,278 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:50,278 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:53:50,280 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:50,280 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:53:50,280 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:53:50,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:53:50,281 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:50,281 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:53:50,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:53:50,281 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:50,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:53:50,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:53:50,282 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:53:50,284 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:53:50,284 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:53:50,284 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:53:50,284 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:53:50,285 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:53:50,285 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:53:50,285 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:53:50,285 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. 2023-04-25 02:53:50,285 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. started... 2023-04-25 02:53:50,285 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. as already flushing 2023-04-25 02:53:50,286 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. 2023-04-25 02:53:50,286 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:53:50,286 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:53:50,286 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:50,286 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:53:50,286 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:53:50,288 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:50,288 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:50,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:50,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:53:50,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:53:50,288 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:50,288 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:53:50,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:53:50,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:53:50,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:50,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:50,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:53:50,290 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:53:50,290 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@15d24778[Count = 0] remaining members to acquire global barrier 2023-04-25 02:53:50,290 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:53:50,290 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:50,292 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:50,292 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:50,292 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:53:50,292 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:50,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:50,292 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:53:50,293 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:53:50,293 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:53:50,295 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:53:50,295 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:53:50,295 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:53:50,295 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:50,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:50,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:53:50,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:53:50,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:53:50,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:53:50,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:50,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:50,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:53:50,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:50,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:50,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:53:50,301 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:53:50,301 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:53:50,301 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:53:50,301 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:53:50,301 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:53:50,301 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:53:50,304 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:50,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:50,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:53:50,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:53:50,304 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:50,304 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:53:50,304 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:50,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:50,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:53:50,305 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:50,305 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:53:50,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:53:50,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:50,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:50,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:53:50,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:50,307 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:50,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:50,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:53:50,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:50,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:50,310 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:50,310 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:53:50,310 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:53:50,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:53:50,310 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:53:50,310 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:53:50,310 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:50,310 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:53:50,311 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:53:50,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:53:50,311 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:50,311 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:50,312 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:50,312 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:53:50,312 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:53:50,312 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:53:50,312 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:53:50,312 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:53:50,313 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:53:50,313 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:53:50,313 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43806 deadline: 1682391290311, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:50,322 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=106 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/.tmp/B/e956d8789b0f4cf497b5c380548a9a4b 2023-04-25 02:53:50,322 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:53:50,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43864 deadline: 1682391290320, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:50,413 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:53:50,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:53:50,417 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:53:50,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:53:50,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:53:50,419 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:53:50,419 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:53:50,420 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:53:50,420 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:53:50,421 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:50,421 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:53:50,424 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:53:50,424 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:53:50,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:53:50,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:53:50,426 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:50,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:53:50,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:53:50,426 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:53:50,427 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:53:50,427 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:53:50,427 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:53:50,427 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:53:50,428 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:50,429 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:53:50,429 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:53:50,429 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:53:50,429 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. 2023-04-25 02:53:50,429 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. started... 2023-04-25 02:53:50,429 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. as already flushing 2023-04-25 02:53:50,429 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. 2023-04-25 02:53:50,430 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:53:50,430 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:53:50,430 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:50,430 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:53:50,430 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:53:50,430 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:50,430 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:53:50,433 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:50,433 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:50,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:50,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:53:50,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:53:50,434 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:50,434 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:53:50,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:53:50,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:53:50,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:50,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:50,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:53:50,436 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:53:50,436 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7153fcf[Count = 0] remaining members to acquire global barrier 2023-04-25 02:53:50,436 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:53:50,436 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:50,437 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:50,437 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:50,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:50,437 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:50,437 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:53:50,437 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:53:50,437 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:53:50,437 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:53:50,440 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:50,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:50,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:53:50,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:53:50,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:53:50,441 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:53:50,441 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:53:50,441 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:53:50,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:53:50,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:50,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:50,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:53:50,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:50,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:50,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:53:50,445 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:53:50,445 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:53:50,445 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:53:50,445 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:53:50,445 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:53:50,445 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:53:50,448 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:50,448 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:53:50,448 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:50,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:50,449 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:53:50,449 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:50,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:53:50,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:50,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:53:50,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:53:50,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:50,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:53:50,450 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:50,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:50,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:53:50,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:50,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:50,457 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:50,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:53:50,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:50,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:50,462 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:50,463 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:53:50,463 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:53:50,463 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:50,463 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:50,463 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:50,463 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:50,462 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:53:50,463 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:53:50,463 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:53:50,463 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:53:50,463 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:53:50,463 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:53:50,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:53:50,465 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:53:50,465 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:53:50,466 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:53:50,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:53:50,545 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:53:50,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43822 deadline: 1682391290542, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:50,566 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:53:50,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:53:50,570 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:53:50,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:53:50,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:53:50,571 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:53:50,571 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:53:50,573 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:53:50,573 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:53:50,573 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:50,573 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:53:50,576 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:50,576 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:53:50,576 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:53:50,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:53:50,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:53:50,576 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:50,577 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:53:50,577 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:50,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:53:50,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:53:50,577 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:53:50,579 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:53:50,579 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:53:50,580 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:53:50,580 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:53:50,581 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:53:50,581 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:53:50,581 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:53:50,581 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. 2023-04-25 02:53:50,581 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. started... 2023-04-25 02:53:50,581 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. as already flushing 2023-04-25 02:53:50,581 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. 2023-04-25 02:53:50,581 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:53:50,581 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:53:50,581 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:50,581 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:53:50,581 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:53:50,583 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:50,583 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:50,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:50,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:53:50,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:53:50,583 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:50,584 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:53:50,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:53:50,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:53:50,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:50,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:50,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:53:50,585 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:53:50,586 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@f429cfe[Count = 0] remaining members to acquire global barrier 2023-04-25 02:53:50,587 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:53:50,587 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:50,592 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:50,592 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:50,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:50,592 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:53:50,592 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:53:50,592 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:53:50,593 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:50,593 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:53:50,594 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:50,594 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:53:50,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:50,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:53:50,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:53:50,594 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:53:50,594 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:53:50,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:53:50,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:53:50,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:50,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:50,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:53:50,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:50,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:50,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:53:50,599 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:53:50,599 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:53:50,599 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:53:50,599 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:53:50,599 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:53:50,599 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:53:50,601 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:50,601 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:53:50,601 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:50,601 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:50,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:50,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:53:50,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:53:50,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:50,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:53:50,602 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:53:50,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:53:50,602 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:50,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:50,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:50,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:53:50,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:50,604 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:50,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:50,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:53:50,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:50,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:50,608 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:53:50,608 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:50,608 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:53:50,608 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:53:50,608 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:53:50,608 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:53:50,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:53:50,608 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:50,608 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:50,608 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:50,608 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:50,608 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:53:50,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:53:50,609 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:53:50,609 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:53:50,608 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:53:50,609 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:53:50,613 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:53:50,609 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:53:50,620 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:53:50,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43806 deadline: 1682391290617, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:50,632 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:53:50,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43864 deadline: 1682391290627, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:50,713 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:53:50,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:53:50,716 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:53:50,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:53:50,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:53:50,718 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:53:50,718 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:53:50,719 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:53:50,719 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:53:50,719 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:50,719 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:53:50,722 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:53:50,722 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:50,722 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:53:50,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:53:50,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:53:50,723 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:50,723 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:53:50,723 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:50,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:53:50,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:53:50,724 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:53:50,744 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:53:50,744 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:53:50,744 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:53:50,744 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:53:50,746 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:53:50,746 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:53:50,746 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:53:50,746 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. 2023-04-25 02:53:50,746 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. started... 2023-04-25 02:53:50,746 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. as already flushing 2023-04-25 02:53:50,747 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. 2023-04-25 02:53:50,747 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:53:50,747 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:53:50,747 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:50,747 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:53:50,747 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:53:50,750 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:50,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:50,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:53:50,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:53:50,750 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:50,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:53:50,751 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:50,751 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:53:50,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:53:50,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:50,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:50,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:53:50,753 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:53:50,753 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@446daddb[Count = 0] remaining members to acquire global barrier 2023-04-25 02:53:50,753 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:53:50,753 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:50,755 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:50,755 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:50,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:50,755 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:53:50,755 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:53:50,755 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:53:50,755 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:50,755 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:53:50,757 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:53:50,757 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:53:50,757 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:53:50,759 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=106 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/.tmp/C/1ee24ddf8c5744828d3deb8cbbf45a91 2023-04-25 02:53:50,761 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:50,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:50,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:53:50,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:53:50,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:53:50,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:53:50,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:50,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:50,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:53:50,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:50,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:50,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:53:50,768 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:53:50,768 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:53:50,768 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:53:50,768 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:53:50,768 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:53:50,768 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:53:50,770 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/.tmp/A/9db33aa5cd3840b2ae0a076d53bc8f0d as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/A/9db33aa5cd3840b2ae0a076d53bc8f0d 2023-04-25 02:53:50,771 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:50,772 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:50,772 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:53:50,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:50,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:53:50,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:53:50,772 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:50,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:50,772 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:53:50,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:53:50,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:50,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:53:50,774 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:50,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:50,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:53:50,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:50,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:50,779 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:50,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:53:50,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:50,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:50,783 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:50,783 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:53:50,783 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:53:50,783 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:50,783 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:50,783 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:50,783 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:50,783 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:53:50,783 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:53:50,783 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:53:50,783 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:53:50,784 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:53:50,784 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:53:50,785 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:53:50,785 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:53:50,785 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:53:50,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:53:50,786 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:53:50,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:53:50,796 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/A/9db33aa5cd3840b2ae0a076d53bc8f0d, entries=200, sequenceid=106, filesize=13.9 K 2023-04-25 02:53:50,805 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/.tmp/B/e956d8789b0f4cf497b5c380548a9a4b as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/B/e956d8789b0f4cf497b5c380548a9a4b 2023-04-25 02:53:50,814 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/B/e956d8789b0f4cf497b5c380548a9a4b, entries=150, sequenceid=106, filesize=11.6 K 2023-04-25 02:53:50,817 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/.tmp/C/1ee24ddf8c5744828d3deb8cbbf45a91 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/C/1ee24ddf8c5744828d3deb8cbbf45a91 2023-04-25 02:53:50,830 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/C/1ee24ddf8c5744828d3deb8cbbf45a91, entries=150, sequenceid=106, filesize=11.6 K 2023-04-25 02:53:50,832 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 047cf722b8f6bff032073f994134764a in 982ms, sequenceid=106, compaction requested=true 2023-04-25 02:53:50,832 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 047cf722b8f6bff032073f994134764a: 2023-04-25 02:53:50,833 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 02:53:50,833 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-25 02:53:50,833 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 02:53:50,833 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-25 02:53:50,833 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 02:53:50,837 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 38101 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-25 02:53:50,838 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HStore(1912): 047cf722b8f6bff032073f994134764a/A is initiating minor compaction (all files) 2023-04-25 02:53:50,838 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 047cf722b8f6bff032073f994134764a/A in TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. 2023-04-25 02:53:50,838 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/A/cc671b2ab2ae4be78ebbe8537bc3c01b, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/A/d0ee89d7f1d143a5a70d112d5ae6fb22, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/A/9db33aa5cd3840b2ae0a076d53bc8f0d] into tmpdir=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/.tmp, totalSize=37.2 K 2023-04-25 02:53:50,839 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35761 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-25 02:53:50,839 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting cc671b2ab2ae4be78ebbe8537bc3c01b, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=60, earliestPutTs=1682391227031 2023-04-25 02:53:50,839 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HStore(1912): 047cf722b8f6bff032073f994134764a/B is initiating minor compaction (all files) 2023-04-25 02:53:50,840 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 047cf722b8f6bff032073f994134764a/B in TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. 2023-04-25 02:53:50,840 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/B/80c60c631b7c454681537f934d1dd2b1, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/B/df13aedd944e410185ba3fadc0fb7e83, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/B/e956d8789b0f4cf497b5c380548a9a4b] into tmpdir=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/.tmp, totalSize=34.9 K 2023-04-25 02:53:50,841 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting d0ee89d7f1d143a5a70d112d5ae6fb22, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=93, earliestPutTs=1682391228064 2023-04-25 02:53:50,841 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting 80c60c631b7c454681537f934d1dd2b1, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=60, earliestPutTs=1682391227031 2023-04-25 02:53:50,842 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting 9db33aa5cd3840b2ae0a076d53bc8f0d, keycount=200, bloomtype=ROW, size=13.9 K, encoding=NONE, compression=NONE, seqNum=106, earliestPutTs=1682391229229 2023-04-25 02:53:50,848 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting df13aedd944e410185ba3fadc0fb7e83, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=93, earliestPutTs=1682391228064 2023-04-25 02:53:50,849 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting e956d8789b0f4cf497b5c380548a9a4b, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=106, earliestPutTs=1682391229229 2023-04-25 02:53:50,864 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] throttle.PressureAwareThroughputController(145): 047cf722b8f6bff032073f994134764a#A#compaction#18 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-25 02:53:50,882 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] throttle.PressureAwareThroughputController(145): 047cf722b8f6bff032073f994134764a#B#compaction#19 average throughput is 1.09 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-25 02:53:50,884 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:53:50,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:53:50,889 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:53:50,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:53:50,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:53:50,902 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:53:50,907 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:53:50,908 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:53:50,908 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:53:50,914 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:50,914 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:53:50,922 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:53:50,922 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:50,922 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:53:50,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:53:50,923 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:50,923 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:53:50,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:53:50,924 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:50,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:53:50,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:53:50,924 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:53:50,925 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:53:50,925 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:53:50,925 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:53:50,925 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:53:50,946 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:53:50,954 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:53:50,954 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:53:50,956 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. 2023-04-25 02:53:50,963 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. started... 2023-04-25 02:53:50,963 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 047cf722b8f6bff032073f994134764a 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-04-25 02:53:50,970 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 047cf722b8f6bff032073f994134764a, store=A 2023-04-25 02:53:50,970 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 02:53:50,970 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 047cf722b8f6bff032073f994134764a, store=B 2023-04-25 02:53:50,970 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 02:53:50,971 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 047cf722b8f6bff032073f994134764a, store=C 2023-04-25 02:53:50,971 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 02:53:50,974 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/.tmp/A/eaaa86ea137f4bd684bfed0cd21d311b as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/A/eaaa86ea137f4bd684bfed0cd21d311b 2023-04-25 02:53:50,985 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 047cf722b8f6bff032073f994134764a/A of 047cf722b8f6bff032073f994134764a into eaaa86ea137f4bd684bfed0cd21d311b(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 02:53:50,986 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 047cf722b8f6bff032073f994134764a: 2023-04-25 02:53:50,986 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a., storeName=047cf722b8f6bff032073f994134764a/A, priority=13, startTime=1682391230833; duration=0sec 2023-04-25 02:53:50,986 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 02:53:50,986 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-25 02:53:50,988 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35761 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-25 02:53:50,988 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HStore(1912): 047cf722b8f6bff032073f994134764a/C is initiating minor compaction (all files) 2023-04-25 02:53:50,988 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 047cf722b8f6bff032073f994134764a/C in TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. 2023-04-25 02:53:50,988 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/C/d978d53d0452462cb593a4a9d336b81a, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/C/79721a49470e41aab987a4934a3f715d, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/C/1ee24ddf8c5744828d3deb8cbbf45a91] into tmpdir=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/.tmp, totalSize=34.9 K 2023-04-25 02:53:50,989 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting d978d53d0452462cb593a4a9d336b81a, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=60, earliestPutTs=1682391227031 2023-04-25 02:53:50,989 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting 79721a49470e41aab987a4934a3f715d, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=93, earliestPutTs=1682391228064 2023-04-25 02:53:50,990 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting 1ee24ddf8c5744828d3deb8cbbf45a91, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=106, earliestPutTs=1682391229229 2023-04-25 02:53:51,009 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] throttle.PressureAwareThroughputController(145): 047cf722b8f6bff032073f994134764a#C#compaction#21 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-25 02:53:51,014 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=143 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/.tmp/A/f8b1717c091841429271f9c3f14e32b5 2023-04-25 02:53:51,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(9158): Flush requested on 047cf722b8f6bff032073f994134764a 2023-04-25 02:53:51,060 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. as already flushing 2023-04-25 02:53:51,095 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:53:51,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43822 deadline: 1682391291091, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:51,125 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:53:51,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43806 deadline: 1682391291125, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:51,140 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:53:51,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43864 deadline: 1682391291136, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:51,197 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:53:51,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43822 deadline: 1682391291197, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:51,379 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/.tmp/B/01bd58c8af354be18d171a3022afad59 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/B/01bd58c8af354be18d171a3022afad59 2023-04-25 02:53:51,398 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 047cf722b8f6bff032073f994134764a/B of 047cf722b8f6bff032073f994134764a into 01bd58c8af354be18d171a3022afad59(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 02:53:51,398 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 047cf722b8f6bff032073f994134764a: 2023-04-25 02:53:51,398 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a., storeName=047cf722b8f6bff032073f994134764a/B, priority=13, startTime=1682391230833; duration=0sec 2023-04-25 02:53:51,399 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 02:53:51,407 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:53:51,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43822 deadline: 1682391291400, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:51,439 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/.tmp/C/f22a298a041a411fa9e6163622766315 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/C/f22a298a041a411fa9e6163622766315 2023-04-25 02:53:51,452 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=143 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/.tmp/B/2fe07082cf584ff5a43ecb56ea7a4271 2023-04-25 02:53:51,453 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 047cf722b8f6bff032073f994134764a/C of 047cf722b8f6bff032073f994134764a into f22a298a041a411fa9e6163622766315(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 02:53:51,453 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 047cf722b8f6bff032073f994134764a: 2023-04-25 02:53:51,453 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a., storeName=047cf722b8f6bff032073f994134764a/C, priority=13, startTime=1682391230833; duration=0sec 2023-04-25 02:53:51,454 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 02:53:51,528 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=143 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/.tmp/C/5192eaba12f14abba2a9d04a287cc410 2023-04-25 02:53:51,550 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/.tmp/A/f8b1717c091841429271f9c3f14e32b5 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/A/f8b1717c091841429271f9c3f14e32b5 2023-04-25 02:53:51,560 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/A/f8b1717c091841429271f9c3f14e32b5, entries=150, sequenceid=143, filesize=11.8 K 2023-04-25 02:53:51,564 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/.tmp/B/2fe07082cf584ff5a43ecb56ea7a4271 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/B/2fe07082cf584ff5a43ecb56ea7a4271 2023-04-25 02:53:51,584 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/B/2fe07082cf584ff5a43ecb56ea7a4271, entries=150, sequenceid=143, filesize=11.8 K 2023-04-25 02:53:51,598 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/.tmp/C/5192eaba12f14abba2a9d04a287cc410 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/C/5192eaba12f14abba2a9d04a287cc410 2023-04-25 02:53:51,619 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/C/5192eaba12f14abba2a9d04a287cc410, entries=150, sequenceid=143, filesize=11.8 K 2023-04-25 02:53:51,621 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=46.96 KB/48090 for 047cf722b8f6bff032073f994134764a in 658ms, sequenceid=143, compaction requested=false 2023-04-25 02:53:51,622 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 047cf722b8f6bff032073f994134764a: 2023-04-25 02:53:51,622 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. 2023-04-25 02:53:51,622 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:53:51,623 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:53:51,623 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:51,623 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:53:51,623 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:53:51,625 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:51,625 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:51,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:51,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:53:51,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:53:51,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:53:51,625 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:51,625 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:53:51,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:53:51,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:51,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:51,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:53:51,627 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:53:51,627 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4383cba9[Count = 0] remaining members to acquire global barrier 2023-04-25 02:53:51,627 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:53:51,627 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:51,629 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:51,629 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:51,629 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:53:51,629 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:51,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:51,630 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:53:51,630 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:53:51,630 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:53:51,631 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:51,631 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:53:51,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:51,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:53:51,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:53:51,631 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:53:51,631 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:53:51,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:53:51,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:53:51,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:51,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:51,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:53:51,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:51,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:51,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:53:51,636 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:53:51,636 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:53:51,636 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:53:51,636 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:53:51,636 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:53:51,636 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:53:51,638 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:51,638 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:53:51,638 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:51,638 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:51,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:51,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:53:51,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:51,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:53:51,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:53:51,639 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:51,639 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:53:51,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:53:51,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:51,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:51,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:53:51,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:51,641 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:51,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:51,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:53:51,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:51,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:51,645 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:51,645 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:53:51,645 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:53:51,645 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:53:51,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:53:51,645 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:53:51,645 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:53:51,646 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:53:51,645 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:53:51,646 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:51,646 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:51,646 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:51,646 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:51,645 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:53:51,647 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:53:51,647 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:53:51,647 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:53:51,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:53:51,649 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:53:51,715 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 047cf722b8f6bff032073f994134764a 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-25 02:53:51,715 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 047cf722b8f6bff032073f994134764a, store=A 2023-04-25 02:53:51,715 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:53:51,715 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 047cf722b8f6bff032073f994134764a, store=B 2023-04-25 02:53:51,716 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:53:51,716 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 047cf722b8f6bff032073f994134764a, store=C 2023-04-25 02:53:51,716 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:53:51,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(9158): Flush requested on 047cf722b8f6bff032073f994134764a 2023-04-25 02:53:51,747 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:53:51,748 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:53:51,751 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:53:51,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:53:51,763 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:53:51,763 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:53:51,763 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:53:51,771 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:53:51,773 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:53:51,774 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:51,774 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:53:51,781 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:53:51,781 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:53:51,781 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:51,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:53:51,785 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:51,785 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:53:51,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:53:51,786 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:51,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:53:51,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:53:51,788 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:53:51,804 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:53:51,804 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:53:51,804 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:53:51,804 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:53:51,809 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:53:51,809 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:53:51,809 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:53:51,809 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=157 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/.tmp/A/a60c550bd51e4ae1877e85b1ebc0036a 2023-04-25 02:53:51,816 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. 2023-04-25 02:53:51,816 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. started... 2023-04-25 02:53:51,816 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. as already flushing 2023-04-25 02:53:51,816 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. 2023-04-25 02:53:51,816 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:53:51,816 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:53:51,816 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:51,816 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:53:51,816 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:53:51,818 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:51,818 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:51,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:51,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:53:51,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:53:51,819 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:51,819 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:53:51,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:53:51,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:53:51,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:51,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:51,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:53:51,825 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:53:51,825 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6d466f51[Count = 0] remaining members to acquire global barrier 2023-04-25 02:53:51,825 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:53:51,825 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:51,827 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:51,827 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:51,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:51,827 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:53:51,827 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:53:51,827 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:53:51,827 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:51,827 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:53:51,829 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:53:51,830 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:53:51,830 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:53:51,830 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:51,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:51,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:53:51,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:53:51,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:53:51,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:53:51,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:51,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:51,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:53:51,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:51,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:51,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:53:51,835 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:53:51,835 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:53:51,835 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:53:51,835 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:53:51,835 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:53:51,835 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:53:51,841 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:51,841 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:53:51,841 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:51,841 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:51,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:51,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:53:51,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:53:51,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:51,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:53:51,842 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:51,848 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:53:51,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:53:51,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:51,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:51,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:53:51,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:51,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:51,850 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:51,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:53:51,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:51,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:51,854 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682391231814, max timestamp=1682391231851Num uniques 859; 2023-04-25 02:53:51,854 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-25 02:53:51,854 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682391231814, max timestamp=1682391231851Num uniques 859; 2023-04-25 02:53:51,854 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682391231814, max timestamp=1682391231851Num uniques 859; 2023-04-25 02:53:51,854 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-25 02:53:51,854 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-25 02:53:51,857 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:53:51,857 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:53:51,857 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:53:51,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:53:51,858 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:51,858 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:53:51,858 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:53:51,858 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:51,858 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:51,858 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:51,858 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:51,858 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:53:51,858 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:53:51,858 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:53:51,860 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:53:51,860 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:53:51,860 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:53:51,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:53:51,936 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:53:51,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43822 deadline: 1682391291933, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:51,960 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:53:51,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:53:51,964 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:53:51,984 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:53:51,984 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:53:51,984 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:53:51,984 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:53:51,985 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:53:51,985 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:53:51,986 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:51,986 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:53:51,988 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:53:51,988 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:51,988 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:53:51,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:53:51,988 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:51,988 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:53:51,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:53:51,989 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:51,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:53:51,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:53:51,990 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:53:52,007 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:53:52,007 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:53:52,007 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:53:52,007 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:53:52,017 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:53:52,017 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:53:52,017 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:53:52,017 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. 2023-04-25 02:53:52,017 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. started... 2023-04-25 02:53:52,017 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. as already flushing 2023-04-25 02:53:52,017 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. 2023-04-25 02:53:52,017 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:53:52,017 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:53:52,017 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:52,018 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:53:52,018 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:53:52,020 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:52,020 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:52,020 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:53:52,024 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:52,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:52,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:53:52,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:53:52,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:53:52,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:53:52,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:52,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:52,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:53:52,027 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:53:52,027 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@76f8ab6b[Count = 0] remaining members to acquire global barrier 2023-04-25 02:53:52,027 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:53:52,027 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:52,029 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:52,029 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:52,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:52,029 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:53:52,029 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:53:52,029 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:53:52,029 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:52,030 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:53:52,031 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:53:52,031 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:53:52,031 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:52,031 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:53:52,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:52,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:53:52,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:53:52,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:53:52,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:53:52,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:52,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:52,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:53:52,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:52,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:52,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:53:52,038 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:53:52,038 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:53:52,038 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:53:52,038 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:53:52,038 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:53:52,038 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:53:52,043 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:52,043 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:53:52,043 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:52,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:52,043 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:52,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:52,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:53:52,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:53:52,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:53:52,044 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:53:52,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:53:52,044 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:52,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:52,045 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:53:52,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43822 deadline: 1682391292039, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:52,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:52,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:53:52,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:52,048 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:52,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:52,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:53:52,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:52,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:52,052 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:53:52,052 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:52,052 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:53:52,052 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:53:52,052 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:53:52,053 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:53:52,052 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:53:52,052 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:53:52,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:53:52,053 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:53:52,057 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:53:52,057 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:52,057 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:52,057 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:52,057 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:52,057 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:53:52,057 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:53:52,057 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:53:52,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:53:52,136 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:53:52,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43806 deadline: 1682391292134, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:52,145 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:53:52,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43864 deadline: 1682391292145, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:52,157 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:53:52,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:53:52,160 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:53:52,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:53:52,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:53:52,162 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:53:52,162 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:53:52,163 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:53:52,163 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:53:52,163 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:52,164 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:53:52,166 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:53:52,166 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:52,166 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:53:52,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:53:52,166 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:52,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:53:52,166 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:53:52,166 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:52,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:53:52,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:53:52,167 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:53:52,167 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:53:52,167 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:53:52,167 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:53:52,167 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:53:52,170 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:53:52,170 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:53:52,170 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:53:52,170 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. 2023-04-25 02:53:52,170 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. started... 2023-04-25 02:53:52,170 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. as already flushing 2023-04-25 02:53:52,170 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. 2023-04-25 02:53:52,170 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:53:52,170 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:53:52,170 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:52,170 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:53:52,170 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:53:52,172 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:52,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:52,172 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:52,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:53:52,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:53:52,173 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:52,173 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:53:52,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:53:52,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:53:52,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:52,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:52,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:53:52,174 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:53:52,174 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5fd2d31a[Count = 0] remaining members to acquire global barrier 2023-04-25 02:53:52,174 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:53:52,174 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:52,176 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:52,176 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:52,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:52,176 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:53:52,176 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:52,176 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:53:52,176 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:53:52,176 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:53:52,178 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:52,178 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:53:52,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:52,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:53:52,178 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:53:52,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:53:52,178 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:53:52,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:53:52,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:53:52,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:52,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:52,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:53:52,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:52,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:52,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:53:52,182 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:53:52,182 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:53:52,182 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:53:52,182 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:53:52,182 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:53:52,182 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:53:52,187 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:52,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:52,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:53:52,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:53:52,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:53:52,188 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:52,188 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:52,188 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:52,188 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:53:52,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:52,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:52,188 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:53:52,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:53:52,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:53:52,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:52,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:52,189 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:52,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:52,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:53:52,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:52,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:52,192 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:52,192 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:53:52,193 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:53:52,193 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:52,193 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:52,193 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:53:52,193 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:52,193 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:53:52,193 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:53:52,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:53:52,193 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:52,193 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:53:52,193 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:53:52,193 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:53:52,193 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:53:52,196 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:53:52,196 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:53:52,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:53:52,196 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:53:52,250 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=157 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/.tmp/B/06ce65c0646b4c498be7edc184a4c269 2023-04-25 02:53:52,252 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:53:52,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43822 deadline: 1682391292248, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:52,264 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:53:52,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43840 deadline: 1682391292264, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:52,269 DEBUG [Thread-147] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4187 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a., hostname=jenkins-hbase4.apache.org,46697,1682391220023, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor28.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-25 02:53:52,280 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:53:52,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43828 deadline: 1682391292278, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:52,281 DEBUG [Thread-145] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4196 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a., hostname=jenkins-hbase4.apache.org,46697,1682391220023, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor28.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-25 02:53:52,298 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:53:52,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:53:52,302 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:53:52,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:53:52,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:53:52,303 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:53:52,303 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:53:52,303 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:53:52,303 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:53:52,303 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:52,303 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:53:52,305 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:52,305 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:53:52,305 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:53:52,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:53:52,305 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:52,305 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:53:52,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:53:52,306 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:52,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:53:52,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:53:52,306 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:53:52,306 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:53:52,306 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:53:52,307 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:53:52,307 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:53:52,307 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:53:52,307 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:53:52,307 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:53:52,307 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. 2023-04-25 02:53:52,307 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. started... 2023-04-25 02:53:52,307 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. as already flushing 2023-04-25 02:53:52,307 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. 2023-04-25 02:53:52,307 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:53:52,307 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:53:52,307 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:52,307 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:53:52,307 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:53:52,309 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:52,309 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:52,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:52,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:53:52,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:53:52,309 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:52,309 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:53:52,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:53:52,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:53:52,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:52,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:52,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:53:52,311 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:53:52,311 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@25185ca[Count = 0] remaining members to acquire global barrier 2023-04-25 02:53:52,311 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:53:52,311 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:52,313 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:52,313 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:52,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:52,313 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:52,313 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:53:52,314 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:53:52,314 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:53:52,314 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:53:52,315 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:53:52,315 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:52,315 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:53:52,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:52,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:53:52,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:53:52,315 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:53:52,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:53:52,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:53:52,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:52,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:52,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:53:52,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:52,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:52,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:53:52,318 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:53:52,318 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:53:52,318 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:53:52,318 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:53:52,318 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:53:52,318 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:53:52,319 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:52,320 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:53:52,320 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:52,320 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:52,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:52,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:53:52,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:53:52,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:52,320 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:53:52,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:53:52,320 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:52,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:53:52,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:52,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:52,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:53:52,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:52,322 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:52,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:52,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:53:52,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:52,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:52,328 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:52,328 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:53:52,328 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:53:52,328 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:53:52,328 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:53:52,328 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:53:52,329 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:53:52,328 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:53:52,328 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:53:52,329 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:53:52,329 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:52,329 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:53:52,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:53:52,329 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:53:52,329 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:52,329 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:52,329 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:52,329 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:53:52,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:53:52,429 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:53:52,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:53:52,432 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:53:52,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:53:52,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:53:52,434 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:53:52,434 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:53:52,434 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:53:52,434 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:53:52,434 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:52,434 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:53:52,436 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:53:52,436 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:52,436 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:53:52,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:53:52,436 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:52,436 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:53:52,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:53:52,437 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:52,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:53:52,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:53:52,437 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:53:52,437 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:53:52,437 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:53:52,438 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:53:52,438 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:53:52,438 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:53:52,438 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:53:52,438 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. 2023-04-25 02:53:52,438 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:53:52,439 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. started... 2023-04-25 02:53:52,439 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. as already flushing 2023-04-25 02:53:52,439 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. 2023-04-25 02:53:52,439 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:53:52,439 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:53:52,439 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:52,439 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:53:52,439 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:53:52,445 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:52,445 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:52,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:52,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:53:52,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:53:52,445 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:52,445 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:53:52,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:53:52,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:53:52,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:52,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:52,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:53:52,447 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:53:52,447 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5c335e36[Count = 0] remaining members to acquire global barrier 2023-04-25 02:53:52,447 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:53:52,447 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:52,448 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:52,448 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:52,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:52,449 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:53:52,449 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:53:52,449 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:52,449 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:53:52,449 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:53:52,453 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:53:52,453 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:53:52,453 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:53:52,458 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:52,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:52,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:53:52,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:53:52,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:53:52,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:53:52,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:52,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:52,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:53:52,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:52,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:52,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:53:52,461 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:53:52,461 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:53:52,461 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:53:52,461 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:53:52,462 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:53:52,462 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:53:52,463 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:52,463 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:53:52,463 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:52,463 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:52,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:52,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:53:52,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:53:52,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:52,465 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:53:52,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:53:52,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:53:52,465 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:52,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:52,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:52,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:53:52,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:52,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:52,466 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:52,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:53:52,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:52,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:52,469 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:53:52,470 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:53:52,470 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:53:52,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:53:52,470 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:53:52,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:53:52,471 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:52,472 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:53:52,472 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:53:52,472 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:52,472 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:53:52,472 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:52,472 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:52,472 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:52,472 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:53:52,472 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:53:52,472 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:53:52,472 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:53:52,472 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:53:52,557 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:53:52,557 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43822 deadline: 1682391292554, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:52,573 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:53:52,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:53:52,575 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:53:52,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:53:52,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:53:52,577 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:53:52,577 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:53:52,577 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:53:52,577 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:53:52,577 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:52,577 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:53:52,579 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:52,579 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:53:52,579 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:53:52,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:53:52,579 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:52,579 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:53:52,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:53:52,580 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:52,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:53:52,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:53:52,580 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:53:52,580 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:53:52,580 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:53:52,580 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:53:52,581 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:53:52,581 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:53:52,587 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:53:52,587 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:53:52,587 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. 2023-04-25 02:53:52,587 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. started... 2023-04-25 02:53:52,587 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. as already flushing 2023-04-25 02:53:52,587 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. 2023-04-25 02:53:52,587 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:53:52,587 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:53:52,587 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:52,587 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:53:52,587 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:53:52,590 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:52,590 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:52,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:52,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:53:52,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:53:52,591 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:52,591 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:53:52,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:53:52,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:53:52,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:52,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:52,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:53:52,592 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:53:52,592 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6f230528[Count = 0] remaining members to acquire global barrier 2023-04-25 02:53:52,592 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:53:52,593 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:52,594 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:52,594 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:52,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:52,594 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:53:52,594 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:53:52,594 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:53:52,594 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:52,594 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:53:52,595 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:53:52,595 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:52,595 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:53:52,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:52,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:53:52,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:53:52,595 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:53:52,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:53:52,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:53:52,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:52,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:52,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:53:52,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:52,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:52,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:53:52,598 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:53:52,598 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:53:52,598 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:53:52,598 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:53:52,598 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:53:52,598 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:53:52,600 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:52,600 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:52,600 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:52,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:52,600 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:53:52,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:52,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:53:52,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:53:52,612 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:53:52,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:53:52,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:53:52,612 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:52,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:52,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:52,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:53:52,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:52,614 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:52,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:52,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:53:52,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:52,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:52,619 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:53:52,619 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:52,619 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:53:52,619 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:53:52,619 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:53:52,619 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:53:52,619 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:53:52,620 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:52,620 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:52,620 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:52,620 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:52,619 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:53:52,619 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:53:52,619 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:53:52,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:53:52,620 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:53:52,620 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:53:52,620 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:53:52,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:53:52,691 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=157 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/.tmp/C/c6473c9403cd427f8dd388cf662b9c8f 2023-04-25 02:53:52,702 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/.tmp/A/a60c550bd51e4ae1877e85b1ebc0036a as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/A/a60c550bd51e4ae1877e85b1ebc0036a 2023-04-25 02:53:52,710 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/A/a60c550bd51e4ae1877e85b1ebc0036a, entries=200, sequenceid=157, filesize=14.1 K 2023-04-25 02:53:52,713 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/.tmp/B/06ce65c0646b4c498be7edc184a4c269 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/B/06ce65c0646b4c498be7edc184a4c269 2023-04-25 02:53:52,720 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:53:52,721 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/B/06ce65c0646b4c498be7edc184a4c269, entries=150, sequenceid=157, filesize=11.8 K 2023-04-25 02:53:52,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:53:52,722 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/.tmp/C/c6473c9403cd427f8dd388cf662b9c8f as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/C/c6473c9403cd427f8dd388cf662b9c8f 2023-04-25 02:53:52,726 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:53:52,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:53:52,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:53:52,729 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:53:52,729 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:53:52,729 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:53:52,729 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:53:52,729 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:52,729 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:53:52,733 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:52,733 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:53:52,733 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:53:52,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:53:52,733 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:52,734 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:53:52,734 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/C/c6473c9403cd427f8dd388cf662b9c8f, entries=150, sequenceid=157, filesize=11.8 K 2023-04-25 02:53:52,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:53:52,736 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:52,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:53:52,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:53:52,736 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 047cf722b8f6bff032073f994134764a in 1021ms, sequenceid=157, compaction requested=true 2023-04-25 02:53:52,736 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:53:52,736 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 047cf722b8f6bff032073f994134764a: 2023-04-25 02:53:52,737 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 02:53:52,737 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 02:53:52,737 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-25 02:53:52,737 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-25 02:53:52,737 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-25 02:53:52,740 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36164 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-25 02:53:52,740 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HStore(1912): 047cf722b8f6bff032073f994134764a/B is initiating minor compaction (all files) 2023-04-25 02:53:52,740 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 047cf722b8f6bff032073f994134764a/B in TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. 2023-04-25 02:53:52,740 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/B/01bd58c8af354be18d171a3022afad59, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/B/2fe07082cf584ff5a43ecb56ea7a4271, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/B/06ce65c0646b4c498be7edc184a4c269] into tmpdir=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/.tmp, totalSize=35.3 K 2023-04-25 02:53:52,740 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 38554 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-25 02:53:52,741 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HStore(1912): 047cf722b8f6bff032073f994134764a/A is initiating minor compaction (all files) 2023-04-25 02:53:52,741 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 047cf722b8f6bff032073f994134764a/A in TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. 2023-04-25 02:53:52,741 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/A/eaaa86ea137f4bd684bfed0cd21d311b, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/A/f8b1717c091841429271f9c3f14e32b5, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/A/a60c550bd51e4ae1877e85b1ebc0036a] into tmpdir=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/.tmp, totalSize=37.7 K 2023-04-25 02:53:52,741 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting eaaa86ea137f4bd684bfed0cd21d311b, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=106, earliestPutTs=1682391229229 2023-04-25 02:53:52,741 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting 01bd58c8af354be18d171a3022afad59, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=106, earliestPutTs=1682391229229 2023-04-25 02:53:52,744 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:53:52,745 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:53:52,745 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:53:52,742 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting f8b1717c091841429271f9c3f14e32b5, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=143, earliestPutTs=1682391229993 2023-04-25 02:53:52,746 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting 2fe07082cf584ff5a43ecb56ea7a4271, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=143, earliestPutTs=1682391229993 2023-04-25 02:53:52,747 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting a60c550bd51e4ae1877e85b1ebc0036a, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=157, earliestPutTs=1682391231075 2023-04-25 02:53:52,747 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting 06ce65c0646b4c498be7edc184a4c269, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=157, earliestPutTs=1682391231075 2023-04-25 02:53:52,748 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:53:52,760 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:53:52,760 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. 2023-04-25 02:53:52,760 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:53:52,760 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. started... 2023-04-25 02:53:52,760 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 047cf722b8f6bff032073f994134764a 3/3 column families, dataSize=127.47 KB heapSize=365.06 KB 2023-04-25 02:53:52,760 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 047cf722b8f6bff032073f994134764a, store=A 2023-04-25 02:53:52,761 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 02:53:52,761 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 047cf722b8f6bff032073f994134764a, store=B 2023-04-25 02:53:52,761 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 02:53:52,761 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 047cf722b8f6bff032073f994134764a, store=C 2023-04-25 02:53:52,761 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 02:53:52,781 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] throttle.PressureAwareThroughputController(145): 047cf722b8f6bff032073f994134764a#B#compaction#27 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-25 02:53:52,798 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] throttle.PressureAwareThroughputController(145): 047cf722b8f6bff032073f994134764a#A#compaction#29 average throughput is 1.64 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-25 02:53:52,851 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/.tmp/A/92e055129a1b4b4e80bd8b072e04aa61 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/A/92e055129a1b4b4e80bd8b072e04aa61 2023-04-25 02:53:52,859 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 047cf722b8f6bff032073f994134764a/A of 047cf722b8f6bff032073f994134764a into 92e055129a1b4b4e80bd8b072e04aa61(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 02:53:52,859 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 047cf722b8f6bff032073f994134764a: 2023-04-25 02:53:52,859 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a., storeName=047cf722b8f6bff032073f994134764a/A, priority=13, startTime=1682391232737; duration=0sec 2023-04-25 02:53:52,859 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 02:53:52,859 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-25 02:53:52,870 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36164 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-25 02:53:52,870 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HStore(1912): 047cf722b8f6bff032073f994134764a/C is initiating minor compaction (all files) 2023-04-25 02:53:52,870 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 047cf722b8f6bff032073f994134764a/C in TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. 2023-04-25 02:53:52,870 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/C/f22a298a041a411fa9e6163622766315, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/C/5192eaba12f14abba2a9d04a287cc410, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/C/c6473c9403cd427f8dd388cf662b9c8f] into tmpdir=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/.tmp, totalSize=35.3 K 2023-04-25 02:53:52,871 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting f22a298a041a411fa9e6163622766315, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=106, earliestPutTs=1682391229229 2023-04-25 02:53:52,871 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting 5192eaba12f14abba2a9d04a287cc410, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=143, earliestPutTs=1682391229993 2023-04-25 02:53:52,872 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting c6473c9403cd427f8dd388cf662b9c8f, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=157, earliestPutTs=1682391231075 2023-04-25 02:53:52,885 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] throttle.PressureAwareThroughputController(145): 047cf722b8f6bff032073f994134764a#C#compaction#30 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-25 02:53:52,917 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/.tmp/C/f305449eff62412ea77f94b8dd919930 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/C/f305449eff62412ea77f94b8dd919930 2023-04-25 02:53:52,928 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 047cf722b8f6bff032073f994134764a/C of 047cf722b8f6bff032073f994134764a into f305449eff62412ea77f94b8dd919930(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 02:53:52,928 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 047cf722b8f6bff032073f994134764a: 2023-04-25 02:53:52,928 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a., storeName=047cf722b8f6bff032073f994134764a/C, priority=13, startTime=1682391232737; duration=0sec 2023-04-25 02:53:52,928 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 02:53:53,064 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. as already flushing 2023-04-25 02:53:53,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(9158): Flush requested on 047cf722b8f6bff032073f994134764a 2023-04-25 02:53:53,105 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:53:53,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43822 deadline: 1682391293102, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:53,208 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:53:53,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43822 deadline: 1682391293207, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:53,223 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=193 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/.tmp/A/32c71762857c40e4b6ba25edc682c6c4 2023-04-25 02:53:53,238 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/.tmp/B/27a973d536f244d2ba53f04fa4895ff7 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/B/27a973d536f244d2ba53f04fa4895ff7 2023-04-25 02:53:53,250 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 047cf722b8f6bff032073f994134764a/B of 047cf722b8f6bff032073f994134764a into 27a973d536f244d2ba53f04fa4895ff7(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 02:53:53,250 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 047cf722b8f6bff032073f994134764a: 2023-04-25 02:53:53,250 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a., storeName=047cf722b8f6bff032073f994134764a/B, priority=13, startTime=1682391232737; duration=0sec 2023-04-25 02:53:53,250 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 02:53:53,264 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=193 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/.tmp/B/5400faad1fa64b0a9f4a926928c5c481 2023-04-25 02:53:53,414 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:53:53,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43822 deadline: 1682391293410, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:53,689 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=193 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/.tmp/C/03f0784e7fb34b3c8345582879f1d6fe 2023-04-25 02:53:53,697 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/.tmp/A/32c71762857c40e4b6ba25edc682c6c4 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/A/32c71762857c40e4b6ba25edc682c6c4 2023-04-25 02:53:53,705 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/A/32c71762857c40e4b6ba25edc682c6c4, entries=150, sequenceid=193, filesize=11.8 K 2023-04-25 02:53:53,708 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/.tmp/B/5400faad1fa64b0a9f4a926928c5c481 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/B/5400faad1fa64b0a9f4a926928c5c481 2023-04-25 02:53:53,717 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:53:53,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43822 deadline: 1682391293716, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:53,719 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/B/5400faad1fa64b0a9f4a926928c5c481, entries=150, sequenceid=193, filesize=11.8 K 2023-04-25 02:53:53,723 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/.tmp/C/03f0784e7fb34b3c8345582879f1d6fe as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/C/03f0784e7fb34b3c8345582879f1d6fe 2023-04-25 02:53:53,730 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/C/03f0784e7fb34b3c8345582879f1d6fe, entries=150, sequenceid=193, filesize=11.8 K 2023-04-25 02:53:53,733 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~365.02 KB/373776, currentSize=53.67 KB/54960 for 047cf722b8f6bff032073f994134764a in 973ms, sequenceid=193, compaction requested=false 2023-04-25 02:53:53,733 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 047cf722b8f6bff032073f994134764a: 2023-04-25 02:53:53,733 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. 2023-04-25 02:53:53,733 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:53:53,734 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:53:53,734 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:53,734 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:53:53,734 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:53:53,736 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:53,736 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:53,736 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:53:53,738 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:53,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:53,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:53:53,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:53:53,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:53:53,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:53:53,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:53,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:53,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:53:53,740 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:53:53,741 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@26a051d5[Count = 0] remaining members to acquire global barrier 2023-04-25 02:53:53,741 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:53:53,741 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:53,742 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:53,742 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:53,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:53,742 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:53:53,742 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:53:53,742 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:53:53,742 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:53,742 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:53:53,744 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:53,744 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:53:53,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:53,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:53:53,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:53:53,744 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:53:53,744 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:53:53,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:53:53,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:53:53,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:53,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:53,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:53:53,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:53,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:53,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:53:53,749 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:53:53,750 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:53:53,750 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:53:53,750 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:53:53,750 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:53:53,750 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:53:53,752 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:53,752 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:53:53,752 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:53,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:53,752 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:53,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:53,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:53:53,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:53:53,753 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:53:53,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:53:53,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:53:53,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:53,754 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:53,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:53:53,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:53,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:53,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:53,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:53:53,761 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:53,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:53,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:53,771 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:53:53,771 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:53,771 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:53:53,771 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:53:53,771 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:53:53,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:53:53,771 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:53:53,771 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:53:53,771 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:53:53,771 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:53:53,772 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:53,772 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:53,772 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:53,772 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:53,772 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:53:53,772 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:53:53,772 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:53:53,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:53:53,872 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:53:53,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:53:53,875 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:53:53,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:53:53,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:53:53,882 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:53:53,882 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:53:53,882 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:53:53,882 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:53:53,883 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:53,883 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:53:53,885 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:53:53,885 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:53,885 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:53:53,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:53:53,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:53:53,886 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:53,886 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:53:53,887 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:53,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:53:53,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:53:53,887 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:53:53,888 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:53:53,888 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:53:53,888 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:53:53,888 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:53:53,888 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:53:53,888 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:53:53,888 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:53:53,888 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. 2023-04-25 02:53:53,888 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. started... 2023-04-25 02:53:53,888 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 047cf722b8f6bff032073f994134764a 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-25 02:53:53,889 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 047cf722b8f6bff032073f994134764a, store=A 2023-04-25 02:53:53,889 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:53:53,889 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 047cf722b8f6bff032073f994134764a, store=B 2023-04-25 02:53:53,889 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:53:53,889 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 047cf722b8f6bff032073f994134764a, store=C 2023-04-25 02:53:53,889 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:53:54,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(9158): Flush requested on 047cf722b8f6bff032073f994134764a 2023-04-25 02:53:54,142 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. as already flushing 2023-04-25 02:53:54,188 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682391234170, max timestamp=1682391234186Num uniques 859; 2023-04-25 02:53:54,188 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-25 02:53:54,189 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682391234170, max timestamp=1682391234186Num uniques 859; 2023-04-25 02:53:54,189 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-25 02:53:54,189 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682391234170, max timestamp=1682391234186Num uniques 859; 2023-04-25 02:53:54,189 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-25 02:53:54,222 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:53:54,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43806 deadline: 1682391294220, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:54,223 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:53:54,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43864 deadline: 1682391294220, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:54,225 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:53:54,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43822 deadline: 1682391294225, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:54,300 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=207 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/.tmp/A/e219f9629b2c44f5985325caecc7cd97 2023-04-25 02:53:54,321 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=207 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/.tmp/B/a43d2baa2ae7496c95a0e390d104b651 2023-04-25 02:53:54,326 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:53:54,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43806 deadline: 1682391294324, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:54,327 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:53:54,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43864 deadline: 1682391294325, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:54,352 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=207 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/.tmp/C/e28cc196ac12477c920283f072ddc5e5 2023-04-25 02:53:54,361 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/.tmp/A/e219f9629b2c44f5985325caecc7cd97 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/A/e219f9629b2c44f5985325caecc7cd97 2023-04-25 02:53:54,367 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/A/e219f9629b2c44f5985325caecc7cd97, entries=150, sequenceid=207, filesize=11.8 K 2023-04-25 02:53:54,372 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/.tmp/B/a43d2baa2ae7496c95a0e390d104b651 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/B/a43d2baa2ae7496c95a0e390d104b651 2023-04-25 02:53:54,379 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/B/a43d2baa2ae7496c95a0e390d104b651, entries=150, sequenceid=207, filesize=11.8 K 2023-04-25 02:53:54,380 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/.tmp/C/e28cc196ac12477c920283f072ddc5e5 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/C/e28cc196ac12477c920283f072ddc5e5 2023-04-25 02:53:54,387 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/C/e28cc196ac12477c920283f072ddc5e5, entries=150, sequenceid=207, filesize=11.8 K 2023-04-25 02:53:54,388 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=134.18 KB/137400 for 047cf722b8f6bff032073f994134764a in 500ms, sequenceid=207, compaction requested=true 2023-04-25 02:53:54,388 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 047cf722b8f6bff032073f994134764a: 2023-04-25 02:53:54,388 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. 2023-04-25 02:53:54,388 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:53:54,389 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:53:54,389 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:54,389 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:53:54,389 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:53:54,395 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:54,395 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:54,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:54,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:53:54,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:53:54,396 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:54,396 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:53:54,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:53:54,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:53:54,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:54,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:54,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:53:54,397 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:53:54,397 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:53:54,397 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2851ff1e[Count = 0] remaining members to acquire global barrier 2023-04-25 02:53:54,398 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:54,399 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:54,399 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:54,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:54,399 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:53:54,399 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:53:54,399 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:53:54,399 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:54,399 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:53:54,400 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:53:54,400 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:54,400 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:53:54,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:54,400 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:53:54,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:53:54,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:53:54,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:53:54,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:53:54,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:54,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:54,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:53:54,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:54,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:54,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:53:54,403 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:53:54,403 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:53:54,403 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:53:54,403 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:53:54,403 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:53:54,403 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:53:54,405 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:54,405 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:54,405 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:54,405 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:53:54,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:54,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:54,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:53:54,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:53:54,406 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:54,406 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:53:54,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:53:54,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:53:54,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:54,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:54,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:53:54,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:54,407 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:54,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:54,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:53:54,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:54,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:54,410 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:54,410 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:53:54,410 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:53:54,410 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:53:54,410 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:53:54,410 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:53:54,410 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:54,410 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:54,410 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:54,410 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:54,410 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:53:54,410 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:53:54,410 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:53:54,411 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:53:54,411 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:53:54,411 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:53:54,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:53:54,411 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:53:54,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:53:54,510 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:53:54,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:53:54,513 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:53:54,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:53:54,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:53:54,515 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:53:54,515 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:53:54,515 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:53:54,515 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:53:54,516 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:54,516 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:53:54,518 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:54,518 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:53:54,519 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:53:54,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:53:54,519 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:54,519 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:53:54,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:53:54,519 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:54,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:53:54,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:53:54,520 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:53:54,520 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:53:54,520 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:53:54,520 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:53:54,520 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:53:54,520 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:53:54,521 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:53:54,521 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:53:54,521 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. 2023-04-25 02:53:54,521 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. started... 2023-04-25 02:53:54,521 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 047cf722b8f6bff032073f994134764a 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-04-25 02:53:54,521 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 047cf722b8f6bff032073f994134764a, store=A 2023-04-25 02:53:54,521 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 02:53:54,522 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 047cf722b8f6bff032073f994134764a, store=B 2023-04-25 02:53:54,522 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 02:53:54,522 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 047cf722b8f6bff032073f994134764a, store=C 2023-04-25 02:53:54,522 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 02:53:54,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(9158): Flush requested on 047cf722b8f6bff032073f994134764a 2023-04-25 02:53:54,532 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. as already flushing 2023-04-25 02:53:54,549 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:53:54,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43864 deadline: 1682391294544, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:54,551 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:53:54,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43806 deadline: 1682391294546, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:54,651 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:53:54,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43864 deadline: 1682391294650, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:54,662 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:53:54,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43806 deadline: 1682391294657, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:54,856 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:53:54,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43864 deadline: 1682391294854, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:54,866 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:53:54,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43806 deadline: 1682391294865, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:54,949 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=244 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/.tmp/A/4dab330a3a2e4a76b62b2e37126a74cf 2023-04-25 02:53:54,972 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=244 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/.tmp/B/85cc0d029a104e25a26fce78c5275423 2023-04-25 02:53:55,010 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=244 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/.tmp/C/cf47055f6bfa4336bba1737336d5fcef 2023-04-25 02:53:55,019 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/.tmp/A/4dab330a3a2e4a76b62b2e37126a74cf as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/A/4dab330a3a2e4a76b62b2e37126a74cf 2023-04-25 02:53:55,027 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/A/4dab330a3a2e4a76b62b2e37126a74cf, entries=150, sequenceid=244, filesize=11.8 K 2023-04-25 02:53:55,029 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/.tmp/B/85cc0d029a104e25a26fce78c5275423 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/B/85cc0d029a104e25a26fce78c5275423 2023-04-25 02:53:55,042 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/B/85cc0d029a104e25a26fce78c5275423, entries=150, sequenceid=244, filesize=11.8 K 2023-04-25 02:53:55,045 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/.tmp/C/cf47055f6bfa4336bba1737336d5fcef as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/C/cf47055f6bfa4336bba1737336d5fcef 2023-04-25 02:53:55,063 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/C/cf47055f6bfa4336bba1737336d5fcef, entries=150, sequenceid=244, filesize=11.8 K 2023-04-25 02:53:55,065 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=46.96 KB/48090 for 047cf722b8f6bff032073f994134764a in 544ms, sequenceid=244, compaction requested=true 2023-04-25 02:53:55,065 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 047cf722b8f6bff032073f994134764a: 2023-04-25 02:53:55,065 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. 2023-04-25 02:53:55,066 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:53:55,067 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:53:55,067 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:55,067 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:53:55,067 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:53:55,082 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:55,082 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:55,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:55,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:53:55,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:53:55,082 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:55,082 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:53:55,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:53:55,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:53:55,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:55,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:55,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:53:55,085 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:53:55,085 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6e55c6cc[Count = 0] remaining members to acquire global barrier 2023-04-25 02:53:55,085 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:53:55,085 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:55,087 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:55,088 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:55,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:55,088 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:53:55,088 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:53:55,088 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:53:55,088 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:55,088 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:53:55,089 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:55,089 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:53:55,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:55,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:53:55,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:53:55,089 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:53:55,090 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:53:55,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:53:55,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:53:55,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:55,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:55,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:53:55,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:55,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:55,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:53:55,092 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:53:55,092 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:53:55,092 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:53:55,092 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:53:55,092 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:53:55,092 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:53:55,094 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:55,094 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:55,094 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:55,094 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:53:55,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:55,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:55,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:53:55,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:53:55,094 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:53:55,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:53:55,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:55,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:53:55,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:55,097 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:55,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:53:55,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:55,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:55,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:53:55,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:55,100 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:55,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:55,102 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:55,102 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:53:55,102 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:53:55,102 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:53:55,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:53:55,102 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:53:55,102 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:53:55,103 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:53:55,103 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:53:55,103 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:55,103 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:55,103 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:55,103 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:55,103 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:53:55,102 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:53:55,103 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:53:55,103 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:53:55,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:53:55,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(9158): Flush requested on 047cf722b8f6bff032073f994134764a 2023-04-25 02:53:55,161 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 047cf722b8f6bff032073f994134764a 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-25 02:53:55,162 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 047cf722b8f6bff032073f994134764a, store=A 2023-04-25 02:53:55,162 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:53:55,162 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 047cf722b8f6bff032073f994134764a, store=B 2023-04-25 02:53:55,162 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:53:55,162 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 047cf722b8f6bff032073f994134764a, store=C 2023-04-25 02:53:55,162 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:53:55,189 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=255 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/.tmp/A/66cc459630d54035a3bef400504e929c 2023-04-25 02:53:55,203 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:53:55,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:53:55,207 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:53:55,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:53:55,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:53:55,216 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:53:55,216 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:53:55,216 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:53:55,216 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:53:55,217 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:55,217 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:53:55,221 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:53:55,221 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:55,221 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:53:55,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:53:55,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:53:55,222 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:55,222 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:53:55,222 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:55,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:53:55,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:53:55,222 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:53:55,223 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:53:55,223 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:53:55,223 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:53:55,223 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:53:55,223 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:53:55,223 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:53:55,223 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:53:55,223 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. 2023-04-25 02:53:55,223 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. started... 2023-04-25 02:53:55,223 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. as already flushing 2023-04-25 02:53:55,223 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. 2023-04-25 02:53:55,223 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:53:55,223 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:53:55,223 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:55,224 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:53:55,224 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:53:55,225 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:55,225 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:55,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:55,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:53:55,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:53:55,225 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:55,226 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:53:55,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:53:55,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:53:55,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:55,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:55,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:53:55,227 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:53:55,227 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@19a8846b[Count = 0] remaining members to acquire global barrier 2023-04-25 02:53:55,227 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:53:55,227 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:55,228 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:55,228 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:55,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:55,229 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:55,229 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:53:55,229 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:53:55,229 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:53:55,229 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:53:55,230 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:53:55,230 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:55,230 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:53:55,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:55,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:53:55,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:53:55,230 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:53:55,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:53:55,231 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682391235215, max timestamp=1682391235230Num uniques 859; 2023-04-25 02:53:55,231 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-25 02:53:55,232 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682391235215, max timestamp=1682391235230Num uniques 859; 2023-04-25 02:53:55,232 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682391235215, max timestamp=1682391235230Num uniques 859; 2023-04-25 02:53:55,232 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-25 02:53:55,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:53:55,232 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-25 02:53:55,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:55,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:55,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:53:55,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:55,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:55,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:53:55,239 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:53:55,239 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:53:55,239 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:53:55,239 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:53:55,239 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:53:55,239 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:53:55,242 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:55,242 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:53:55,242 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:55,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:55,242 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:53:55,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:53:55,243 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:55,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:55,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:53:55,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:53:55,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:55,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:53:55,244 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:55,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:55,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:53:55,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:55,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:55,246 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:55,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:53:55,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:55,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:55,249 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:55,249 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:53:55,249 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:53:55,249 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:53:55,249 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:53:55,249 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:53:55,249 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:53:55,249 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:55,249 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:53:55,249 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:53:55,249 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:55,249 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:53:55,250 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:53:55,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:53:55,250 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:55,250 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:55,250 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:53:55,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:53:55,264 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:53:55,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43822 deadline: 1682391295260, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:55,265 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:53:55,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43864 deadline: 1682391295262, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:55,268 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:53:55,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43806 deadline: 1682391295264, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:55,350 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:53:55,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:53:55,354 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:53:55,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:53:55,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:53:55,356 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:53:55,356 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:53:55,356 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:53:55,356 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:53:55,357 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:55,357 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:53:55,358 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:55,358 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:55,359 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:53:55,361 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:53:55,361 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:53:55,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:53:55,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:53:55,363 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:55,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:53:55,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:53:55,363 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:53:55,369 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:53:55,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43822 deadline: 1682391295366, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:55,369 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:53:55,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43864 deadline: 1682391295367, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:55,372 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:53:55,372 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:53:55,372 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:53:55,372 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:53:55,372 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:53:55,373 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:53:55,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43806 deadline: 1682391295370, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:55,384 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:53:55,385 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:53:55,385 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. 2023-04-25 02:53:55,385 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. started... 2023-04-25 02:53:55,385 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. as already flushing 2023-04-25 02:53:55,385 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. 2023-04-25 02:53:55,385 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:53:55,385 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:53:55,385 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:55,385 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:53:55,385 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:53:55,394 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:55,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:55,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:53:55,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:53:55,394 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:55,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:53:55,395 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:55,395 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:53:55,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:53:55,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:55,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:55,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:53:55,399 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:53:55,399 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3da24072[Count = 0] remaining members to acquire global barrier 2023-04-25 02:53:55,399 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:53:55,399 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:55,404 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:55,404 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:55,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:55,404 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:53:55,404 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:53:55,404 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:53:55,405 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:55,405 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:53:55,406 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:53:55,406 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:55,406 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:53:55,406 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:53:55,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:55,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:53:55,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:53:55,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:53:55,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:53:55,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:55,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:55,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:53:55,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:55,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:55,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:53:55,410 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:53:55,410 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:53:55,410 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:53:55,410 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:53:55,410 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:53:55,410 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:53:55,412 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:55,412 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:55,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:55,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:53:55,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:53:55,412 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:55,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:55,412 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:53:55,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:53:55,413 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:53:55,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:53:55,413 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:55,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:55,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:55,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:53:55,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:55,414 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:55,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:55,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:53:55,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:55,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:55,421 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:55,421 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:53:55,421 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:53:55,421 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:53:55,421 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:53:55,421 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:53:55,421 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:53:55,422 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:53:55,421 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:53:55,421 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:55,422 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:55,422 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:55,422 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:55,422 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:53:55,422 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:53:55,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:53:55,423 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:53:55,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:53:55,522 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:53:55,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:53:55,552 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:53:55,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:53:55,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:53:55,559 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:53:55,559 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:53:55,559 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:53:55,559 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:53:55,559 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:55,560 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:53:55,561 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:53:55,561 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:55,561 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:53:55,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:53:55,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:53:55,562 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:55,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:53:55,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:53:55,563 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:53:55,563 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:55,563 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:53:55,565 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:53:55,565 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:53:55,565 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:53:55,565 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:53:55,566 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:53:55,566 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:53:55,566 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. 2023-04-25 02:53:55,566 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. started... 2023-04-25 02:53:55,566 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. as already flushing 2023-04-25 02:53:55,566 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. 2023-04-25 02:53:55,566 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:53:55,566 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:53:55,567 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:55,567 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:53:55,567 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:53:55,569 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:55,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:55,570 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:55,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:53:55,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:53:55,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:53:55,571 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:55,571 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:53:55,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:53:55,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:55,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:55,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:53:55,573 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:53:55,573 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1a2cf1d7[Count = 0] remaining members to acquire global barrier 2023-04-25 02:53:55,573 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:53:55,573 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:55,574 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:53:55,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43822 deadline: 1682391295571, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:55,574 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:55,574 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:55,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:55,575 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:53:55,575 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:53:55,575 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:53:55,575 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:53:55,575 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:55,575 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:53:55,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43864 deadline: 1682391295573, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:55,578 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:53:55,578 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:53:55,578 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:53:55,578 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:55,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:55,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:53:55,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:53:55,579 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:53:55,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43806 deadline: 1682391295578, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:55,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:53:55,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:53:55,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:55,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:55,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:53:55,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:55,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:55,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:53:55,588 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:53:55,588 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:53:55,588 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:53:55,588 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:53:55,588 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:53:55,588 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:53:55,592 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:55,592 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:53:55,592 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:55,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:55,592 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:55,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:55,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:53:55,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:53:55,592 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:53:55,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:53:55,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:53:55,593 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:55,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:55,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:55,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:53:55,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:55,595 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:55,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:55,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:53:55,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:55,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:55,598 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:53:55,598 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:53:55,598 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:55,598 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:53:55,598 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:53:55,598 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:55,598 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:55,598 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:55,598 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:55,598 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:53:55,599 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:53:55,599 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:53:55,599 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:53:55,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:53:55,599 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:53:55,599 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:53:55,602 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:53:55,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:53:55,602 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:53:55,625 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=255 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/.tmp/B/6b8cced2b09542a18ef2dacac211e4b3 2023-04-25 02:53:55,650 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=255 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/.tmp/C/813081adb8a9484a895c1cbbbb341063 2023-04-25 02:53:55,665 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/.tmp/A/66cc459630d54035a3bef400504e929c as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/A/66cc459630d54035a3bef400504e929c 2023-04-25 02:53:55,672 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/A/66cc459630d54035a3bef400504e929c, entries=150, sequenceid=255, filesize=11.8 K 2023-04-25 02:53:55,674 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/.tmp/B/6b8cced2b09542a18ef2dacac211e4b3 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/B/6b8cced2b09542a18ef2dacac211e4b3 2023-04-25 02:53:55,680 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/B/6b8cced2b09542a18ef2dacac211e4b3, entries=150, sequenceid=255, filesize=11.8 K 2023-04-25 02:53:55,682 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/.tmp/C/813081adb8a9484a895c1cbbbb341063 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/C/813081adb8a9484a895c1cbbbb341063 2023-04-25 02:53:55,688 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/C/813081adb8a9484a895c1cbbbb341063, entries=150, sequenceid=255, filesize=11.8 K 2023-04-25 02:53:55,689 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 047cf722b8f6bff032073f994134764a in 528ms, sequenceid=255, compaction requested=true 2023-04-25 02:53:55,689 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 047cf722b8f6bff032073f994134764a: 2023-04-25 02:53:55,689 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 02:53:55,689 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 02:53:55,689 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-25 02:53:55,689 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-25 02:53:55,689 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-25 02:53:55,692 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 60488 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-25 02:53:55,693 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HStore(1912): 047cf722b8f6bff032073f994134764a/A is initiating minor compaction (all files) 2023-04-25 02:53:55,693 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 047cf722b8f6bff032073f994134764a/A in TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. 2023-04-25 02:53:55,693 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/A/92e055129a1b4b4e80bd8b072e04aa61, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/A/32c71762857c40e4b6ba25edc682c6c4, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/A/e219f9629b2c44f5985325caecc7cd97, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/A/4dab330a3a2e4a76b62b2e37126a74cf, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/A/66cc459630d54035a3bef400504e929c] into tmpdir=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/.tmp, totalSize=59.1 K 2023-04-25 02:53:55,699 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:53:55,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:53:55,705 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 60488 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-25 02:53:55,705 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HStore(1912): 047cf722b8f6bff032073f994134764a/B is initiating minor compaction (all files) 2023-04-25 02:53:55,706 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 047cf722b8f6bff032073f994134764a/B in TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. 2023-04-25 02:53:55,706 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/B/27a973d536f244d2ba53f04fa4895ff7, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/B/5400faad1fa64b0a9f4a926928c5c481, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/B/a43d2baa2ae7496c95a0e390d104b651, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/B/85cc0d029a104e25a26fce78c5275423, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/B/6b8cced2b09542a18ef2dacac211e4b3] into tmpdir=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/.tmp, totalSize=59.1 K 2023-04-25 02:53:55,706 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting 92e055129a1b4b4e80bd8b072e04aa61, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=157, earliestPutTs=1682391231075 2023-04-25 02:53:55,707 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting 27a973d536f244d2ba53f04fa4895ff7, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=157, earliestPutTs=1682391231075 2023-04-25 02:53:55,708 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:53:55,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:53:55,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:53:55,710 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:53:55,710 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:53:55,710 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:53:55,710 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:53:55,710 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:55,710 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:53:55,712 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting 32c71762857c40e4b6ba25edc682c6c4, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=193, earliestPutTs=1682391231920 2023-04-25 02:53:55,712 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:53:55,712 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:53:55,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:53:55,712 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting 5400faad1fa64b0a9f4a926928c5c481, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=193, earliestPutTs=1682391231920 2023-04-25 02:53:55,713 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting e219f9629b2c44f5985325caecc7cd97, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=207, earliestPutTs=1682391233080 2023-04-25 02:53:55,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:53:55,713 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:55,713 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting 4dab330a3a2e4a76b62b2e37126a74cf, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=244, earliestPutTs=1682391234196 2023-04-25 02:53:55,713 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:55,714 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:55,714 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:53:55,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:53:55,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:53:55,714 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:53:55,714 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting a43d2baa2ae7496c95a0e390d104b651, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=207, earliestPutTs=1682391233080 2023-04-25 02:53:55,714 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting 66cc459630d54035a3bef400504e929c, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=255, earliestPutTs=1682391234543 2023-04-25 02:53:55,715 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting 85cc0d029a104e25a26fce78c5275423, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=244, earliestPutTs=1682391234196 2023-04-25 02:53:55,716 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting 6b8cced2b09542a18ef2dacac211e4b3, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=255, earliestPutTs=1682391234543 2023-04-25 02:53:55,716 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:53:55,716 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:53:55,716 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:53:55,716 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:53:55,717 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:53:55,728 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:53:55,728 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:53:55,728 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. 2023-04-25 02:53:55,728 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. started... 2023-04-25 02:53:55,728 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 047cf722b8f6bff032073f994134764a 3/3 column families, dataSize=127.47 KB heapSize=365.06 KB 2023-04-25 02:53:55,729 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 047cf722b8f6bff032073f994134764a, store=A 2023-04-25 02:53:55,729 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 02:53:55,729 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 047cf722b8f6bff032073f994134764a, store=B 2023-04-25 02:53:55,729 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 02:53:55,729 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 047cf722b8f6bff032073f994134764a, store=C 2023-04-25 02:53:55,729 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 02:53:55,731 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] throttle.PressureAwareThroughputController(145): 047cf722b8f6bff032073f994134764a#A#compaction#42 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-25 02:53:55,754 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] throttle.PressureAwareThroughputController(145): 047cf722b8f6bff032073f994134764a#B#compaction#43 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-25 02:53:55,804 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/.tmp/A/2b90aa83e9e24e1898b7def9a4e87a69 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/A/2b90aa83e9e24e1898b7def9a4e87a69 2023-04-25 02:53:55,807 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/.tmp/B/a09959b0235e46eca0ae302da0c1b63c as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/B/a09959b0235e46eca0ae302da0c1b63c 2023-04-25 02:53:55,814 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 047cf722b8f6bff032073f994134764a/A of 047cf722b8f6bff032073f994134764a into 2b90aa83e9e24e1898b7def9a4e87a69(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 02:53:55,814 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 047cf722b8f6bff032073f994134764a: 2023-04-25 02:53:55,814 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a., storeName=047cf722b8f6bff032073f994134764a/A, priority=11, startTime=1682391235689; duration=0sec 2023-04-25 02:53:55,814 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 02:53:55,814 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-25 02:53:55,815 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 047cf722b8f6bff032073f994134764a/B of 047cf722b8f6bff032073f994134764a into a09959b0235e46eca0ae302da0c1b63c(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 02:53:55,816 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 047cf722b8f6bff032073f994134764a: 2023-04-25 02:53:55,816 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a., storeName=047cf722b8f6bff032073f994134764a/B, priority=11, startTime=1682391235689; duration=0sec 2023-04-25 02:53:55,816 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 02:53:55,817 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 60488 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-25 02:53:55,817 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HStore(1912): 047cf722b8f6bff032073f994134764a/C is initiating minor compaction (all files) 2023-04-25 02:53:55,817 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 047cf722b8f6bff032073f994134764a/C in TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. 2023-04-25 02:53:55,817 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/C/f305449eff62412ea77f94b8dd919930, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/C/03f0784e7fb34b3c8345582879f1d6fe, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/C/e28cc196ac12477c920283f072ddc5e5, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/C/cf47055f6bfa4336bba1737336d5fcef, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/C/813081adb8a9484a895c1cbbbb341063] into tmpdir=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/.tmp, totalSize=59.1 K 2023-04-25 02:53:55,818 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting f305449eff62412ea77f94b8dd919930, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=157, earliestPutTs=1682391231075 2023-04-25 02:53:55,818 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting 03f0784e7fb34b3c8345582879f1d6fe, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=193, earliestPutTs=1682391231920 2023-04-25 02:53:55,829 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting e28cc196ac12477c920283f072ddc5e5, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=207, earliestPutTs=1682391233080 2023-04-25 02:53:55,830 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting cf47055f6bfa4336bba1737336d5fcef, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=244, earliestPutTs=1682391234196 2023-04-25 02:53:55,831 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting 813081adb8a9484a895c1cbbbb341063, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=255, earliestPutTs=1682391234543 2023-04-25 02:53:55,862 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] throttle.PressureAwareThroughputController(145): 047cf722b8f6bff032073f994134764a#C#compaction#45 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-25 02:53:55,881 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. as already flushing 2023-04-25 02:53:55,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(9158): Flush requested on 047cf722b8f6bff032073f994134764a 2023-04-25 02:53:55,887 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/.tmp/C/5827a2ca106c4a58b0e61b09103fe534 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/C/5827a2ca106c4a58b0e61b09103fe534 2023-04-25 02:53:55,919 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 047cf722b8f6bff032073f994134764a/C of 047cf722b8f6bff032073f994134764a into 5827a2ca106c4a58b0e61b09103fe534(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 02:53:55,919 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 047cf722b8f6bff032073f994134764a: 2023-04-25 02:53:55,919 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a., storeName=047cf722b8f6bff032073f994134764a/C, priority=11, startTime=1682391235689; duration=0sec 2023-04-25 02:53:55,919 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 02:53:55,934 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:53:55,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43864 deadline: 1682391295931, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:55,941 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:53:55,941 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:53:55,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43806 deadline: 1682391295935, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:55,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43822 deadline: 1682391295935, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:56,038 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:53:56,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43864 deadline: 1682391296036, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:56,044 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:53:56,044 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43806 deadline: 1682391296043, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:56,044 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:53:56,044 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43822 deadline: 1682391296043, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:56,202 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=291 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/.tmp/A/f42105b99bed441c900da64221720246 2023-04-25 02:53:56,243 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:53:56,243 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43864 deadline: 1682391296241, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:56,248 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:53:56,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43806 deadline: 1682391296247, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:56,248 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:53:56,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43822 deadline: 1682391296247, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:56,304 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:53:56,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43840 deadline: 1682391296302, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:56,306 DEBUG [Thread-147] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8227 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a., hostname=jenkins-hbase4.apache.org,46697,1682391220023, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor28.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-25 02:53:56,316 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:53:56,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43828 deadline: 1682391296315, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:56,317 DEBUG [Thread-145] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8232 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a., hostname=jenkins-hbase4.apache.org,46697,1682391220023, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor28.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-25 02:53:56,549 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:53:56,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43864 deadline: 1682391296546, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:56,554 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:53:56,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43822 deadline: 1682391296550, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:56,554 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:53:56,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43806 deadline: 1682391296552, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:56,619 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=291 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/.tmp/B/f2c22b8fae3e473995c520ed366ba64d 2023-04-25 02:53:57,058 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:53:57,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43864 deadline: 1682391297054, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:57,059 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:53:57,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43806 deadline: 1682391297057, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:57,061 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:53:57,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43822 deadline: 1682391297060, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:57,063 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=291 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/.tmp/C/b887bfeb411c4be6ad60af38c202355c 2023-04-25 02:53:57,074 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/.tmp/A/f42105b99bed441c900da64221720246 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/A/f42105b99bed441c900da64221720246 2023-04-25 02:53:57,082 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/A/f42105b99bed441c900da64221720246, entries=150, sequenceid=291, filesize=11.9 K 2023-04-25 02:53:57,084 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/.tmp/B/f2c22b8fae3e473995c520ed366ba64d as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/B/f2c22b8fae3e473995c520ed366ba64d 2023-04-25 02:53:57,093 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/B/f2c22b8fae3e473995c520ed366ba64d, entries=150, sequenceid=291, filesize=11.9 K 2023-04-25 02:53:57,095 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/.tmp/C/b887bfeb411c4be6ad60af38c202355c as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/C/b887bfeb411c4be6ad60af38c202355c 2023-04-25 02:53:57,103 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/C/b887bfeb411c4be6ad60af38c202355c, entries=150, sequenceid=291, filesize=11.9 K 2023-04-25 02:53:57,104 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~365.02 KB/373776, currentSize=60.38 KB/61830 for 047cf722b8f6bff032073f994134764a in 1376ms, sequenceid=291, compaction requested=false 2023-04-25 02:53:57,104 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 047cf722b8f6bff032073f994134764a: 2023-04-25 02:53:57,105 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. 2023-04-25 02:53:57,105 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:53:57,110 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:53:57,110 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:57,110 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:53:57,110 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:53:57,151 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:57,151 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:57,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:57,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:53:57,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:53:57,151 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:57,151 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:53:57,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:53:57,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:53:57,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:57,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:57,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:53:57,154 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:53:57,154 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5a8f27f8[Count = 0] remaining members to acquire global barrier 2023-04-25 02:53:57,154 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:53:57,154 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:57,213 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:57,213 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:57,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:57,213 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:57,213 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:53:57,213 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:53:57,213 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:53:57,214 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:53:57,258 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:57,258 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:53:57,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:57,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:53:57,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:53:57,259 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:53:57,259 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:53:57,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:53:57,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:53:57,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:57,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:57,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:53:57,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:57,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:57,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:53:57,261 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:53:57,261 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:53:57,262 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:53:57,262 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:53:57,262 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:53:57,262 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:53:57,312 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:57,312 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:57,313 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:53:57,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:57,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:53:57,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:53:57,313 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:57,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:57,313 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:53:57,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:53:57,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:53:57,314 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:57,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:57,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:57,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:53:57,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:57,316 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:57,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:57,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:53:57,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:57,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:57,343 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:57,343 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:53:57,343 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:53:57,343 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:53:57,343 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:57,343 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:57,343 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:57,343 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:57,343 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:53:57,343 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:53:57,344 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:53:57,343 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:53:57,344 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:53:57,344 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:53:57,344 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:53:57,344 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:53:57,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:53:57,345 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:53:57,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:53:57,444 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:53:57,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:53:57,447 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:53:57,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:53:57,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:53:57,458 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:53:57,458 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:53:57,458 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:53:57,458 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:53:57,459 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:57,459 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:53:57,520 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:57,520 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:53:57,520 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:53:57,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:53:57,521 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:57,521 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:53:57,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:53:57,522 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:57,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:53:57,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:53:57,523 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:53:57,525 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:53:57,526 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:53:57,526 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:53:57,526 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:53:57,526 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:53:57,526 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:53:57,526 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:53:57,526 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. 2023-04-25 02:53:57,526 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. started... 2023-04-25 02:53:57,526 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 047cf722b8f6bff032073f994134764a 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-25 02:53:57,527 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 047cf722b8f6bff032073f994134764a, store=A 2023-04-25 02:53:57,527 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:53:57,527 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 047cf722b8f6bff032073f994134764a, store=B 2023-04-25 02:53:57,527 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:53:57,527 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 047cf722b8f6bff032073f994134764a, store=C 2023-04-25 02:53:57,527 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:53:57,949 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=306 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/.tmp/A/09cc91e9cdd44c12bba2e7c3bebe89df 2023-04-25 02:53:57,970 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=306 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/.tmp/B/d64320d988474929a0088055f27703f6 2023-04-25 02:53:58,068 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. as already flushing 2023-04-25 02:53:58,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(9158): Flush requested on 047cf722b8f6bff032073f994134764a 2023-04-25 02:53:58,106 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682391238093, max timestamp=1682391238101Num uniques 859; 2023-04-25 02:53:58,106 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-25 02:53:58,106 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682391238093, max timestamp=1682391238101Num uniques 859; 2023-04-25 02:53:58,106 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-25 02:53:58,107 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682391238093, max timestamp=1682391238101Num uniques 859; 2023-04-25 02:53:58,107 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-25 02:53:58,147 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:53:58,149 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:53:58,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43806 deadline: 1682391298140, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:58,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43822 deadline: 1682391298138, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:58,152 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:53:58,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43864 deadline: 1682391298144, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:58,253 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:53:58,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43806 deadline: 1682391298251, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:58,255 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:53:58,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43822 deadline: 1682391298253, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:58,258 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:53:58,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43864 deadline: 1682391298255, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:58,390 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=306 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/.tmp/C/5bb31b7ca4204cbdb08a05d2a9c69c3e 2023-04-25 02:53:58,398 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/.tmp/A/09cc91e9cdd44c12bba2e7c3bebe89df as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/A/09cc91e9cdd44c12bba2e7c3bebe89df 2023-04-25 02:53:58,414 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/A/09cc91e9cdd44c12bba2e7c3bebe89df, entries=150, sequenceid=306, filesize=11.9 K 2023-04-25 02:53:58,415 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/.tmp/B/d64320d988474929a0088055f27703f6 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/B/d64320d988474929a0088055f27703f6 2023-04-25 02:53:58,425 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/B/d64320d988474929a0088055f27703f6, entries=150, sequenceid=306, filesize=11.9 K 2023-04-25 02:53:58,426 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/.tmp/C/5bb31b7ca4204cbdb08a05d2a9c69c3e as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/C/5bb31b7ca4204cbdb08a05d2a9c69c3e 2023-04-25 02:53:58,434 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/C/5bb31b7ca4204cbdb08a05d2a9c69c3e, entries=150, sequenceid=306, filesize=11.9 K 2023-04-25 02:53:58,437 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=120.76 KB/123660 for 047cf722b8f6bff032073f994134764a in 911ms, sequenceid=306, compaction requested=true 2023-04-25 02:53:58,437 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 047cf722b8f6bff032073f994134764a: 2023-04-25 02:53:58,437 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. 2023-04-25 02:53:58,437 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:53:58,437 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:53:58,437 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:58,437 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:53:58,437 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:53:58,480 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:58,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:58,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:53:58,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:53:58,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:53:58,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:53:58,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(9158): Flush requested on 047cf722b8f6bff032073f994134764a 2023-04-25 02:53:58,482 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 047cf722b8f6bff032073f994134764a 3/3 column families, dataSize=127.47 KB heapSize=355.69 KB 2023-04-25 02:53:58,482 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 047cf722b8f6bff032073f994134764a, store=A 2023-04-25 02:53:58,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:58,482 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 02:53:58,482 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 047cf722b8f6bff032073f994134764a, store=B 2023-04-25 02:53:58,482 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 02:53:58,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:58,482 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 047cf722b8f6bff032073f994134764a, store=C 2023-04-25 02:53:58,482 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 02:53:58,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:53:58,484 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:53:58,484 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7d97b1e[Count = 0] remaining members to acquire global barrier 2023-04-25 02:53:58,484 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:53:58,484 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:58,485 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:58,501 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:53:58,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43864 deadline: 1682391298497, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:58,505 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=340 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/.tmp/A/4d00fb6ac870456a8d76d669c1102d68 2023-04-25 02:53:58,505 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:53:58,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43806 deadline: 1682391298502, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:58,507 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:53:58,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43822 deadline: 1682391298505, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:58,607 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:53:58,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43864 deadline: 1682391298605, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:58,609 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:53:58,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43806 deadline: 1682391298608, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:58,614 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:53:58,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43822 deadline: 1682391298614, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:58,632 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(162): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:58,632 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:58,632 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:53:58,632 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:53:58,632 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:53:58,632 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:53:58,633 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:58,633 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:53:58,674 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:58,674 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:53:58,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:58,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:53:58,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:53:58,674 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:53:58,674 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:53:58,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:53:58,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:53:58,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:58,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:58,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:53:58,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:58,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:58,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:53:58,677 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:53:58,677 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:53:58,677 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:53:58,677 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:53:58,677 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:53:58,677 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:53:58,751 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:58,751 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:58,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:58,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:53:58,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:53:58,751 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:58,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:58,751 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:53:58,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:53:58,752 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:53:58,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:53:58,752 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:58,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:58,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:58,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:53:58,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:58,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:58,754 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:58,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:53:58,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:58,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:58,773 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:58,774 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:53:58,774 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:53:58,774 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:58,774 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:53:58,774 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:53:58,774 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:53:58,774 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:53:58,774 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:58,774 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:58,774 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:53:58,774 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:58,774 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:53:58,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:53:58,774 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:58,774 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:53:58,774 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:53:58,775 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:53:58,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:53:58,813 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:53:58,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43864 deadline: 1682391298810, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:58,814 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:53:58,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43806 deadline: 1682391298811, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:58,816 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:53:58,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43822 deadline: 1682391298815, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:58,875 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:53:58,875 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:53:58,878 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:53:58,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:53:58,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:53:58,881 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:53:58,881 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:53:58,881 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:53:58,881 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:53:58,881 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:58,882 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:53:58,883 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:58,883 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:58,883 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:53:58,884 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:53:58,884 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:53:58,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:53:58,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:53:58,887 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:58,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:53:58,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:53:58,887 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:53:58,888 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:53:58,888 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:53:58,888 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:53:58,888 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:53:58,888 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:53:58,888 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:53:58,888 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. 2023-04-25 02:53:58,888 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:53:58,888 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. started... 2023-04-25 02:53:58,889 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. as already flushing 2023-04-25 02:53:58,890 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. 2023-04-25 02:53:58,890 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:53:58,890 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:53:58,890 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:58,890 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:53:58,890 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:53:58,892 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:58,892 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:58,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:58,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:53:58,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:53:58,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:53:58,892 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:58,892 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:53:58,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:53:58,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:58,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:58,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:53:58,894 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:53:58,894 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@28d71a47[Count = 0] remaining members to acquire global barrier 2023-04-25 02:53:58,894 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:53:58,894 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:58,895 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:58,896 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:58,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:58,896 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:53:58,896 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:53:58,896 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:53:58,897 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:58,897 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:53:58,899 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:53:58,899 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:53:58,899 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:53:58,899 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:58,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:58,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:53:58,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:53:58,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:53:58,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:53:58,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:58,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:58,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:53:58,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:58,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:58,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:53:58,903 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:53:58,903 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:53:58,903 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:53:58,903 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:53:58,904 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:53:58,904 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:53:58,905 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:58,905 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:58,906 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:58,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:58,905 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:53:58,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:58,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:53:58,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:53:58,906 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:53:58,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:53:58,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:53:58,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:58,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:58,908 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:58,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:53:58,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:58,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:58,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:53:58,909 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:58,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:58,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:58,914 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:58,914 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:53:58,914 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:53:58,915 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:53:58,915 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:53:58,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:53:58,915 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:53:58,915 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:58,915 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:58,915 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:58,915 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:58,915 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:53:58,915 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:53:58,915 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:53:58,915 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:53:58,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:53:58,915 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:53:58,915 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:53:58,926 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=340 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/.tmp/B/148dc052a953472382a76365c9c783e2 2023-04-25 02:53:59,016 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:53:59,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:53:59,018 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:53:59,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:53:59,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:53:59,020 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:53:59,020 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:53:59,020 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:53:59,020 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:53:59,021 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:59,021 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:53:59,022 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:53:59,022 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:59,022 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:53:59,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:53:59,022 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:59,022 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:53:59,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:53:59,023 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:59,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:53:59,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:53:59,023 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:53:59,024 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:53:59,024 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:53:59,024 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:53:59,024 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:53:59,024 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:53:59,024 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:53:59,024 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:53:59,024 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. 2023-04-25 02:53:59,024 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. started... 2023-04-25 02:53:59,024 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. as already flushing 2023-04-25 02:53:59,024 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. 2023-04-25 02:53:59,025 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:53:59,025 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:53:59,025 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:59,025 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:53:59,025 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:53:59,030 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:59,030 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:59,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:59,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:53:59,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:53:59,031 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:59,031 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:53:59,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:53:59,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:53:59,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:59,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:59,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:53:59,033 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:53:59,033 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5588f4c3[Count = 0] remaining members to acquire global barrier 2023-04-25 02:53:59,033 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:53:59,033 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:59,034 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:59,034 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:59,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:59,034 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:53:59,034 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:53:59,034 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:53:59,034 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:59,034 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:53:59,036 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:53:59,036 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:59,036 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:53:59,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:59,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:53:59,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:53:59,036 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:53:59,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:53:59,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:53:59,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:59,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:59,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:53:59,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:59,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:59,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:53:59,040 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:53:59,040 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:53:59,040 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:53:59,040 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:53:59,040 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:53:59,040 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:53:59,043 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:59,043 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:59,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:59,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:53:59,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:53:59,043 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:59,043 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:53:59,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:59,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:53:59,046 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:53:59,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:53:59,047 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:59,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:59,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:59,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:53:59,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:59,048 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:59,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:59,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:53:59,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:59,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:59,052 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:59,052 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:53:59,052 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:53:59,052 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:53:59,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:53:59,052 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:53:59,052 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:53:59,052 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:59,052 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:59,052 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:59,052 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:59,052 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:53:59,052 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:53:59,052 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:53:59,052 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:53:59,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:53:59,052 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:53:59,053 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:53:59,053 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:53:59,116 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:53:59,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43864 deadline: 1682391299116, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:59,117 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:53:59,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43806 deadline: 1682391299116, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:59,118 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:53:59,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43822 deadline: 1682391299117, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:59,153 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:53:59,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:53:59,156 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:53:59,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:53:59,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:53:59,168 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:53:59,168 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:53:59,168 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:53:59,168 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:53:59,169 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:59,169 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:53:59,173 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:59,173 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:53:59,173 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:53:59,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:53:59,173 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:59,174 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:53:59,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:53:59,174 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:59,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:53:59,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:53:59,175 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:53:59,176 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:53:59,176 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:53:59,176 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:53:59,176 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:53:59,176 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:53:59,180 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:53:59,181 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:53:59,181 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. 2023-04-25 02:53:59,181 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. started... 2023-04-25 02:53:59,181 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. as already flushing 2023-04-25 02:53:59,181 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. 2023-04-25 02:53:59,181 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:53:59,181 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:53:59,181 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:59,181 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:53:59,181 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:53:59,183 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:59,183 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:59,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:59,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:53:59,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:53:59,183 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:59,183 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:53:59,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:53:59,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:53:59,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:59,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:59,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:53:59,185 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:53:59,185 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@761bb96d[Count = 0] remaining members to acquire global barrier 2023-04-25 02:53:59,185 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:53:59,186 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:59,187 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:59,187 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:53:59,188 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:59,188 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:59,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:59,188 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:53:59,188 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:53:59,188 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:53:59,193 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:59,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:59,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:53:59,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:53:59,194 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:53:59,194 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:53:59,194 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:53:59,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:53:59,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:53:59,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:59,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:59,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:53:59,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:59,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:59,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:53:59,197 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:53:59,197 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:53:59,197 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:53:59,197 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:53:59,197 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:53:59,197 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:53:59,198 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:59,198 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:59,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:59,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:53:59,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:53:59,198 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:59,198 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:53:59,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:59,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:53:59,199 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:53:59,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:53:59,199 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:59,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:59,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:59,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:53:59,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:59,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:59,200 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:59,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:53:59,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:59,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:59,203 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:59,203 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:53:59,203 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:53:59,203 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:53:59,203 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:53:59,203 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:53:59,203 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:53:59,203 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:53:59,204 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:53:59,203 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:59,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:53:59,204 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:59,204 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:59,204 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:59,204 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:53:59,204 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:53:59,204 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:53:59,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:53:59,304 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:53:59,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:53:59,307 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:53:59,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:53:59,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:53:59,308 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:53:59,308 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:53:59,308 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:53:59,308 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:53:59,309 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:59,309 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:53:59,310 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:53:59,310 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:59,310 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:53:59,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:53:59,311 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:59,311 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:53:59,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:53:59,312 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:59,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:53:59,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:53:59,313 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:53:59,313 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:53:59,313 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:53:59,313 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:53:59,313 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:53:59,313 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:53:59,313 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:53:59,313 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:53:59,314 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. 2023-04-25 02:53:59,314 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. started... 2023-04-25 02:53:59,314 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. as already flushing 2023-04-25 02:53:59,314 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. 2023-04-25 02:53:59,314 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:53:59,314 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:53:59,314 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:59,314 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:53:59,314 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:53:59,317 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:59,317 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:59,317 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:53:59,328 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:59,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:59,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:53:59,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:53:59,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:53:59,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:53:59,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:59,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:59,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:53:59,330 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:53:59,330 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3a562d3b[Count = 0] remaining members to acquire global barrier 2023-04-25 02:53:59,330 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:53:59,330 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:59,332 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:59,332 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:59,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:59,332 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:53:59,332 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:53:59,332 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:53:59,332 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:59,332 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:53:59,334 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:59,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:59,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:53:59,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:53:59,334 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:53:59,340 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:53:59,340 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:53:59,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:53:59,341 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=340 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/.tmp/C/d843da56122c4c96b30e528f6d37dbc8 2023-04-25 02:53:59,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:53:59,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:59,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:59,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:53:59,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:59,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:59,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:53:59,345 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:53:59,345 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:53:59,346 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:53:59,346 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:53:59,346 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:53:59,346 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:53:59,348 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:59,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:59,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:53:59,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:53:59,348 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:59,348 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:53:59,348 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:59,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:59,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:53:59,349 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:59,349 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:53:59,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:53:59,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:59,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:59,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:53:59,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:53:59,352 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:59,353 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/.tmp/A/4d00fb6ac870456a8d76d669c1102d68 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/A/4d00fb6ac870456a8d76d669c1102d68 2023-04-25 02:53:59,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:59,356 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:53:59,356 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:53:59,356 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:53:59,357 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:53:59,357 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:53:59,359 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:53:59,360 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:53:59,360 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:53:59,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:53:59,360 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:59,360 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:53:59,360 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:53:59,360 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:59,360 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:59,360 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:53:59,360 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:59,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:53:59,361 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:53:59,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:53:59,361 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:53:59,370 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/A/4d00fb6ac870456a8d76d669c1102d68, entries=200, sequenceid=340, filesize=14.3 K 2023-04-25 02:53:59,372 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/.tmp/B/148dc052a953472382a76365c9c783e2 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/B/148dc052a953472382a76365c9c783e2 2023-04-25 02:53:59,380 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/B/148dc052a953472382a76365c9c783e2, entries=150, sequenceid=340, filesize=11.9 K 2023-04-25 02:53:59,381 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/.tmp/C/d843da56122c4c96b30e528f6d37dbc8 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/C/d843da56122c4c96b30e528f6d37dbc8 2023-04-25 02:53:59,387 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/C/d843da56122c4c96b30e528f6d37dbc8, entries=150, sequenceid=340, filesize=11.9 K 2023-04-25 02:53:59,388 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~355.64 KB/364176, currentSize=60.38 KB/61830 for 047cf722b8f6bff032073f994134764a in 907ms, sequenceid=340, compaction requested=true 2023-04-25 02:53:59,388 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 047cf722b8f6bff032073f994134764a: 2023-04-25 02:53:59,389 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 02:53:59,389 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-25 02:53:59,389 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-25 02:53:59,389 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 02:53:59,389 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 02:53:59,391 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 51512 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-25 02:53:59,391 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HStore(1912): 047cf722b8f6bff032073f994134764a/A is initiating minor compaction (all files) 2023-04-25 02:53:59,391 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 047cf722b8f6bff032073f994134764a/A in TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. 2023-04-25 02:53:59,391 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/A/2b90aa83e9e24e1898b7def9a4e87a69, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/A/f42105b99bed441c900da64221720246, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/A/09cc91e9cdd44c12bba2e7c3bebe89df, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/A/4d00fb6ac870456a8d76d669c1102d68] into tmpdir=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/.tmp, totalSize=50.3 K 2023-04-25 02:53:59,391 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49072 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-25 02:53:59,391 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HStore(1912): 047cf722b8f6bff032073f994134764a/B is initiating minor compaction (all files) 2023-04-25 02:53:59,391 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 047cf722b8f6bff032073f994134764a/B in TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. 2023-04-25 02:53:59,392 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/B/a09959b0235e46eca0ae302da0c1b63c, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/B/f2c22b8fae3e473995c520ed366ba64d, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/B/d64320d988474929a0088055f27703f6, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/B/148dc052a953472382a76365c9c783e2] into tmpdir=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/.tmp, totalSize=47.9 K 2023-04-25 02:53:59,392 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting 2b90aa83e9e24e1898b7def9a4e87a69, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=255, earliestPutTs=1682391234543 2023-04-25 02:53:59,393 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting f42105b99bed441c900da64221720246, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=291, earliestPutTs=1682391235258 2023-04-25 02:53:59,393 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting a09959b0235e46eca0ae302da0c1b63c, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=255, earliestPutTs=1682391234543 2023-04-25 02:53:59,393 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting f2c22b8fae3e473995c520ed366ba64d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=291, earliestPutTs=1682391235258 2023-04-25 02:53:59,394 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting 09cc91e9cdd44c12bba2e7c3bebe89df, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=306, earliestPutTs=1682391235927 2023-04-25 02:53:59,394 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting d64320d988474929a0088055f27703f6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=306, earliestPutTs=1682391235927 2023-04-25 02:53:59,394 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting 4d00fb6ac870456a8d76d669c1102d68, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=340, earliestPutTs=1682391238129 2023-04-25 02:53:59,396 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting 148dc052a953472382a76365c9c783e2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=340, earliestPutTs=1682391238134 2023-04-25 02:53:59,430 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] throttle.PressureAwareThroughputController(145): 047cf722b8f6bff032073f994134764a#B#compaction#54 average throughput is 1.09 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-25 02:53:59,431 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] throttle.PressureAwareThroughputController(145): 047cf722b8f6bff032073f994134764a#A#compaction#55 average throughput is 0.94 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-25 02:53:59,457 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:53:59,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:53:59,460 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:53:59,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:53:59,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:53:59,468 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:53:59,468 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:53:59,468 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:53:59,468 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:53:59,469 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:59,469 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:53:59,470 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:53:59,470 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:53:59,470 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:59,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:53:59,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:53:59,471 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:59,471 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:53:59,472 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:53:59,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:53:59,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:53:59,473 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:53:59,473 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:53:59,473 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:53:59,474 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/.tmp/A/d8c8b27ce5b4407b83635cde50552811 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/A/d8c8b27ce5b4407b83635cde50552811 2023-04-25 02:53:59,474 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:53:59,474 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:53:59,474 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:53:59,476 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:53:59,476 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:53:59,476 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. 2023-04-25 02:53:59,476 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. started... 2023-04-25 02:53:59,476 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 047cf722b8f6bff032073f994134764a 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-25 02:53:59,476 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 047cf722b8f6bff032073f994134764a, store=A 2023-04-25 02:53:59,477 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:53:59,477 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 047cf722b8f6bff032073f994134764a, store=B 2023-04-25 02:53:59,477 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:53:59,477 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 047cf722b8f6bff032073f994134764a, store=C 2023-04-25 02:53:59,477 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:53:59,482 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 047cf722b8f6bff032073f994134764a/A of 047cf722b8f6bff032073f994134764a into d8c8b27ce5b4407b83635cde50552811(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 02:53:59,482 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 047cf722b8f6bff032073f994134764a: 2023-04-25 02:53:59,482 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a., storeName=047cf722b8f6bff032073f994134764a/A, priority=12, startTime=1682391239388; duration=0sec 2023-04-25 02:53:59,482 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 02:53:59,482 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-25 02:53:59,484 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49072 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-25 02:53:59,484 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HStore(1912): 047cf722b8f6bff032073f994134764a/C is initiating minor compaction (all files) 2023-04-25 02:53:59,484 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 047cf722b8f6bff032073f994134764a/C in TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. 2023-04-25 02:53:59,485 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/C/5827a2ca106c4a58b0e61b09103fe534, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/C/b887bfeb411c4be6ad60af38c202355c, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/C/5bb31b7ca4204cbdb08a05d2a9c69c3e, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/C/d843da56122c4c96b30e528f6d37dbc8] into tmpdir=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/.tmp, totalSize=47.9 K 2023-04-25 02:53:59,488 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting 5827a2ca106c4a58b0e61b09103fe534, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=255, earliestPutTs=1682391234543 2023-04-25 02:53:59,489 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting b887bfeb411c4be6ad60af38c202355c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=291, earliestPutTs=1682391235258 2023-04-25 02:53:59,489 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting 5bb31b7ca4204cbdb08a05d2a9c69c3e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=306, earliestPutTs=1682391235927 2023-04-25 02:53:59,490 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting d843da56122c4c96b30e528f6d37dbc8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=340, earliestPutTs=1682391238134 2023-04-25 02:53:59,491 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=352 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/.tmp/A/b9669c3bf81348e8a2f28215cedb5d65 2023-04-25 02:53:59,506 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] throttle.PressureAwareThroughputController(145): 047cf722b8f6bff032073f994134764a#C#compaction#57 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-25 02:53:59,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(9158): Flush requested on 047cf722b8f6bff032073f994134764a 2023-04-25 02:53:59,621 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. as already flushing 2023-04-25 02:53:59,647 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:53:59,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43864 deadline: 1682391299643, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:59,652 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:53:59,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43806 deadline: 1682391299648, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:59,652 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:53:59,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43822 deadline: 1682391299648, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:59,750 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:53:59,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43864 deadline: 1682391299749, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:59,755 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:53:59,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43822 deadline: 1682391299754, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:59,755 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:53:59,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43806 deadline: 1682391299754, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:59,874 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/.tmp/B/a1cc178c0a8a4cc1ab0b069edcea4299 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/B/a1cc178c0a8a4cc1ab0b069edcea4299 2023-04-25 02:53:59,881 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 047cf722b8f6bff032073f994134764a/B of 047cf722b8f6bff032073f994134764a into a1cc178c0a8a4cc1ab0b069edcea4299(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 02:53:59,881 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 047cf722b8f6bff032073f994134764a: 2023-04-25 02:53:59,881 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a., storeName=047cf722b8f6bff032073f994134764a/B, priority=12, startTime=1682391239389; duration=0sec 2023-04-25 02:53:59,881 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 02:53:59,934 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=352 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/.tmp/B/47603300674344efab19e82c9c31815e 2023-04-25 02:53:59,942 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/.tmp/C/1abffb9b26b44eaabb092b01c9237253 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/C/1abffb9b26b44eaabb092b01c9237253 2023-04-25 02:53:59,953 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:53:59,953 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43864 deadline: 1682391299952, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:59,958 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 047cf722b8f6bff032073f994134764a/C of 047cf722b8f6bff032073f994134764a into 1abffb9b26b44eaabb092b01c9237253(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 02:53:59,958 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 047cf722b8f6bff032073f994134764a: 2023-04-25 02:53:59,958 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a., storeName=047cf722b8f6bff032073f994134764a/C, priority=12, startTime=1682391239389; duration=0sec 2023-04-25 02:53:59,958 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 02:53:59,961 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:53:59,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43806 deadline: 1682391299957, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:59,962 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:53:59,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43822 deadline: 1682391299958, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:53:59,982 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=352 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/.tmp/C/3642c7d23d994d8faf26b213d870e59e 2023-04-25 02:53:59,990 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/.tmp/A/b9669c3bf81348e8a2f28215cedb5d65 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/A/b9669c3bf81348e8a2f28215cedb5d65 2023-04-25 02:53:59,996 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/A/b9669c3bf81348e8a2f28215cedb5d65, entries=100, sequenceid=352, filesize=9.5 K 2023-04-25 02:53:59,998 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/.tmp/B/47603300674344efab19e82c9c31815e as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/B/47603300674344efab19e82c9c31815e 2023-04-25 02:54:00,005 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/B/47603300674344efab19e82c9c31815e, entries=100, sequenceid=352, filesize=9.5 K 2023-04-25 02:54:00,006 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/.tmp/C/3642c7d23d994d8faf26b213d870e59e as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/C/3642c7d23d994d8faf26b213d870e59e 2023-04-25 02:54:00,019 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/C/3642c7d23d994d8faf26b213d870e59e, entries=100, sequenceid=352, filesize=9.5 K 2023-04-25 02:54:00,021 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 047cf722b8f6bff032073f994134764a in 545ms, sequenceid=352, compaction requested=false 2023-04-25 02:54:00,021 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 047cf722b8f6bff032073f994134764a: 2023-04-25 02:54:00,021 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. 2023-04-25 02:54:00,021 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:54:00,022 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:54:00,022 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:00,022 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:54:00,022 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:54:00,024 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:00,024 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:00,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:00,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:00,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:00,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:00,025 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:00,025 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:54:00,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:00,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:00,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:00,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:00,027 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:54:00,027 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2ac31d14[Count = 0] remaining members to acquire global barrier 2023-04-25 02:54:00,027 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:54:00,027 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:00,029 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:00,029 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:00,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:00,029 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:54:00,029 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:54:00,029 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:54:00,029 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:00,029 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:54:00,033 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:54:00,033 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:00,033 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:00,033 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:54:00,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:00,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:00,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:00,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:00,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:00,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:00,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:00,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:00,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:00,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:00,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:54:00,037 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:54:00,037 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:54:00,037 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:54:00,037 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:54:00,037 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:54:00,037 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:54:00,040 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:00,040 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:00,040 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:00,040 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:00,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:00,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:00,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:00,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:00,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:00,040 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:00,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:00,041 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:00,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:00,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:00,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:00,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:00,042 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:00,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:00,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:00,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:00,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:00,045 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:00,045 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:00,045 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:00,045 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:00,045 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:00,045 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:00,045 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:00,045 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:54:00,045 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:54:00,045 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:54:00,045 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:00,046 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:54:00,046 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:54:00,046 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:00,046 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:00,046 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:00,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:00,048 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:00,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:00,146 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:54:00,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:54:00,150 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:54:00,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:54:00,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:54:00,152 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:54:00,152 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:00,152 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:54:00,152 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:54:00,153 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:00,153 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:00,154 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:00,154 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:00,154 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:00,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:00,154 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:00,154 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:54:00,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:00,155 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:00,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:54:00,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:00,155 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:54:00,156 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:54:00,156 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:54:00,156 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:54:00,156 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:00,156 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:54:00,156 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:54:00,156 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:54:00,156 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. 2023-04-25 02:54:00,156 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. started... 2023-04-25 02:54:00,156 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 047cf722b8f6bff032073f994134764a 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-25 02:54:00,157 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 047cf722b8f6bff032073f994134764a, store=A 2023-04-25 02:54:00,157 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:54:00,157 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 047cf722b8f6bff032073f994134764a, store=B 2023-04-25 02:54:00,157 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:54:00,157 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 047cf722b8f6bff032073f994134764a, store=C 2023-04-25 02:54:00,157 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:54:00,176 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=375 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/.tmp/A/111876c02cfd48da805b4ee7cfc7579e 2023-04-25 02:54:00,191 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=375 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/.tmp/B/4bef52e5bba34900a2691e99883fd41b 2023-04-25 02:54:00,233 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=375 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/.tmp/C/e55d362edac649419466f512c3babd31 2023-04-25 02:54:00,242 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/.tmp/A/111876c02cfd48da805b4ee7cfc7579e as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/A/111876c02cfd48da805b4ee7cfc7579e 2023-04-25 02:54:00,251 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/A/111876c02cfd48da805b4ee7cfc7579e, entries=150, sequenceid=375, filesize=11.9 K 2023-04-25 02:54:00,253 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/.tmp/B/4bef52e5bba34900a2691e99883fd41b as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/B/4bef52e5bba34900a2691e99883fd41b 2023-04-25 02:54:00,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(9158): Flush requested on 047cf722b8f6bff032073f994134764a 2023-04-25 02:54:00,268 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. as already flushing 2023-04-25 02:54:00,281 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/B/4bef52e5bba34900a2691e99883fd41b, entries=150, sequenceid=375, filesize=11.9 K 2023-04-25 02:54:00,282 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/.tmp/C/e55d362edac649419466f512c3babd31 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/C/e55d362edac649419466f512c3babd31 2023-04-25 02:54:00,287 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:00,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43864 deadline: 1682391300284, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:00,288 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:00,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43822 deadline: 1682391300286, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:00,289 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:00,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43806 deadline: 1682391300287, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:00,290 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/C/e55d362edac649419466f512c3babd31, entries=150, sequenceid=375, filesize=11.9 K 2023-04-25 02:54:00,291 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 047cf722b8f6bff032073f994134764a in 135ms, sequenceid=375, compaction requested=true 2023-04-25 02:54:00,291 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 047cf722b8f6bff032073f994134764a: 2023-04-25 02:54:00,291 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. 2023-04-25 02:54:00,291 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:54:00,292 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:54:00,292 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:00,292 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:54:00,292 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:54:00,294 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:00,294 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:00,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:00,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:00,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:00,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:00,295 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:00,295 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:54:00,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:00,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:00,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:00,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:00,296 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:54:00,296 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@387fda63[Count = 0] remaining members to acquire global barrier 2023-04-25 02:54:00,296 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:54:00,296 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:00,298 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:00,298 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:00,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:00,298 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:54:00,298 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:00,298 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:54:00,298 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:54:00,298 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:54:00,304 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:54:00,304 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:00,304 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:54:00,304 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:00,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:00,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:00,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:00,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:00,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:00,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:00,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:00,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:00,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:00,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:00,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:54:00,307 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:54:00,307 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:54:00,308 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:54:00,308 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:54:00,308 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:54:00,308 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:54:00,310 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:00,310 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:00,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:00,310 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:00,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:00,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:00,310 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:00,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:00,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:00,311 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:00,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:00,311 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:00,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:00,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:00,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:00,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:00,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:00,312 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:00,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:00,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:00,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:00,318 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:00,318 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:00,318 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:00,318 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:54:00,318 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:00,318 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:54:00,318 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:54:00,318 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:00,318 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:00,318 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:00,318 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:00,318 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:00,318 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:00,318 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:00,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:00,318 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:54:00,318 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:54:00,319 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:00,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:00,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(9158): Flush requested on 047cf722b8f6bff032073f994134764a 2023-04-25 02:54:00,400 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 047cf722b8f6bff032073f994134764a 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-25 02:54:00,401 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 047cf722b8f6bff032073f994134764a, store=A 2023-04-25 02:54:00,401 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:54:00,401 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 047cf722b8f6bff032073f994134764a, store=B 2023-04-25 02:54:00,401 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:54:00,401 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 047cf722b8f6bff032073f994134764a, store=C 2023-04-25 02:54:00,401 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:54:00,419 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:54:00,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:54:00,421 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:54:00,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:54:00,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:54:00,436 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:54:00,436 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:00,436 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:54:00,436 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:54:00,437 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:00,437 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:00,437 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:00,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43806 deadline: 1682391300435, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:00,439 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:00,439 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:00,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:00,439 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:00,440 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:00,440 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:54:00,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:00,440 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:00,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:54:00,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:00,441 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:54:00,441 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:00,441 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=389 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/.tmp/A/fb6b7f329f11485b8c4e77d0daf433e0 2023-04-25 02:54:00,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43822 deadline: 1682391300436, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:00,444 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:54:00,444 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:54:00,444 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:54:00,444 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:00,444 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:54:00,444 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:54:00,444 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:54:00,444 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. 2023-04-25 02:54:00,444 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. started... 2023-04-25 02:54:00,444 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:00,445 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. as already flushing 2023-04-25 02:54:00,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43864 deadline: 1682391300437, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:00,445 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. 2023-04-25 02:54:00,445 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:54:00,445 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:54:00,445 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:00,445 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:54:00,445 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:54:00,447 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:00,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:00,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:00,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:00,447 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:00,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:00,448 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:00,448 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:54:00,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:00,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:00,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:00,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:00,452 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:54:00,452 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1384ccd0[Count = 0] remaining members to acquire global barrier 2023-04-25 02:54:00,452 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:54:00,452 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:00,453 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:00,453 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:54:00,454 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:00,454 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:00,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:00,454 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:54:00,454 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:54:00,454 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:54:00,457 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:00,457 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:54:00,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:00,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:00,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:00,457 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:00,457 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:54:00,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:00,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:00,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:00,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:00,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:00,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:00,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:00,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:54:00,461 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:54:00,461 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:54:00,461 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:54:00,461 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:54:00,461 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:54:00,461 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:54:00,462 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:00,463 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:00,463 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:00,463 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:00,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:00,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:00,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:00,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:00,463 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:00,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:00,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:00,463 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:00,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:00,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:00,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:00,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:00,465 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:00,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:00,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:00,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:00,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:00,468 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:00,468 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:00,468 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:00,468 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:00,468 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:00,468 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:54:00,468 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:00,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:00,468 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:54:00,468 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:54:00,468 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:00,469 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:00,469 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:00,469 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:00,469 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:00,469 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:54:00,469 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:54:00,469 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:00,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:00,540 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:00,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43806 deadline: 1682391300539, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:00,544 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:00,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43822 deadline: 1682391300543, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:00,551 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:00,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43864 deadline: 1682391300546, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:00,569 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:54:00,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:54:00,571 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:54:00,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:54:00,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:54:00,576 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:54:00,576 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:00,576 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:54:00,576 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:54:00,576 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:00,576 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:00,578 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:00,578 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:00,578 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:00,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:00,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:00,579 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:00,579 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:54:00,579 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:00,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:54:00,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:00,580 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:54:00,580 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:54:00,580 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:54:00,580 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:54:00,580 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:00,580 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:54:00,580 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:54:00,580 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:54:00,580 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. 2023-04-25 02:54:00,581 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. started... 2023-04-25 02:54:00,581 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. as already flushing 2023-04-25 02:54:00,581 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. 2023-04-25 02:54:00,581 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:54:00,581 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:54:00,581 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:00,581 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:54:00,581 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:54:00,584 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:00,584 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:00,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:00,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:00,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:00,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:00,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:00,585 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:00,585 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:54:00,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:00,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:00,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:00,586 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:54:00,586 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@18b0146e[Count = 0] remaining members to acquire global barrier 2023-04-25 02:54:00,586 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:54:00,586 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:00,587 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:00,587 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:00,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:00,587 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:54:00,587 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:54:00,587 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:54:00,590 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:54:00,590 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:00,590 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:54:00,594 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:00,594 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:54:00,594 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:54:00,594 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:54:00,594 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:54:00,594 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:54:00,594 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:54:00,594 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:54:00,596 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:00,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:00,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:00,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:00,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:00,598 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:00,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:00,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:00,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:00,599 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:00,599 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:00,599 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:00,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:00,600 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:00,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:00,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:00,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:00,600 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:00,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:00,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:00,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:00,604 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:00,604 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:00,604 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:00,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:00,604 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:54:00,604 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:54:00,604 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:00,604 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:00,604 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:00,605 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:00,605 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:00,605 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:54:00,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:00,605 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:54:00,605 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:00,605 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:00,605 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:00,605 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:00,705 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:54:00,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:54:00,707 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:54:00,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:54:00,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:54:00,708 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:54:00,708 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:00,709 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:54:00,709 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:54:00,709 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:00,709 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:00,710 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:00,710 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:00,710 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:00,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:00,711 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:00,711 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:54:00,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:00,711 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:00,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:54:00,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:00,712 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:54:00,712 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:54:00,712 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:54:00,712 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:54:00,712 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:00,712 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:54:00,712 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:54:00,712 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:54:00,712 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. 2023-04-25 02:54:00,712 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. started... 2023-04-25 02:54:00,713 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. as already flushing 2023-04-25 02:54:00,713 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. 2023-04-25 02:54:00,713 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:54:00,713 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:54:00,713 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:00,713 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:54:00,713 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:54:00,714 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:00,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:00,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:00,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:00,715 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:00,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:00,716 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:00,716 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:54:00,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:00,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:00,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:00,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:00,717 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:54:00,717 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2aa991d1[Count = 0] remaining members to acquire global barrier 2023-04-25 02:54:00,717 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:54:00,717 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:00,718 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:00,718 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:00,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:00,718 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:54:00,719 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:54:00,719 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:54:00,719 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:00,719 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:54:00,721 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:00,721 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:54:00,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:00,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:00,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:00,721 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:00,722 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:54:00,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:00,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:00,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:00,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:00,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:00,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:00,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:00,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:54:00,724 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:54:00,724 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:54:00,724 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:54:00,724 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:54:00,724 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:54:00,724 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:54:00,726 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:00,726 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:00,726 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:00,726 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:00,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:00,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:00,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:00,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:00,727 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:00,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:00,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:00,727 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:00,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:00,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:00,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:00,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:00,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:00,729 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:00,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:00,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:00,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:00,736 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:00,736 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:00,736 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:00,736 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:00,736 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:00,736 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:00,736 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:00,736 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:00,736 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:00,736 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:00,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:00,740 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:00,740 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:54:00,740 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:54:00,740 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:00,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:00,740 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:54:00,740 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:54:00,743 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:00,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43806 deadline: 1682391300742, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:00,749 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:00,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43822 deadline: 1682391300746, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:00,755 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:00,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43864 deadline: 1682391300754, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:00,840 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:54:00,841 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:54:00,844 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:54:00,845 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:54:00,845 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:54:00,845 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:54:00,845 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:00,845 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:54:00,845 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:54:00,846 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:00,846 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:00,849 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:00,849 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:00,849 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:00,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:00,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:00,849 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:00,849 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:54:00,851 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:00,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:54:00,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:00,851 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:54:00,851 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:54:00,852 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:54:00,852 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:54:00,852 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:00,852 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:54:00,852 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:54:00,852 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:54:00,852 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. 2023-04-25 02:54:00,852 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. started... 2023-04-25 02:54:00,852 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. as already flushing 2023-04-25 02:54:00,852 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. 2023-04-25 02:54:00,852 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:54:00,852 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:54:00,852 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:00,852 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:54:00,852 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:54:00,854 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:00,854 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:00,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:00,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:00,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:00,854 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:00,854 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:54:00,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:00,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:00,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:00,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:00,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:00,856 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:54:00,856 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6c43ea25[Count = 0] remaining members to acquire global barrier 2023-04-25 02:54:00,856 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:54:00,856 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:00,857 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:00,857 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:00,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:00,857 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:54:00,857 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:54:00,857 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:00,857 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:54:00,857 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:54:00,858 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:00,858 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:54:00,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:00,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:00,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:00,859 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:00,859 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:54:00,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:00,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:00,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:00,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:00,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:00,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:00,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:00,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:54:00,862 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:54:00,862 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:54:00,862 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:54:00,862 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:54:00,862 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:54:00,862 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:54:00,865 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:00,865 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:00,865 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:00,865 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:00,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:00,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:00,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:00,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:00,865 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:00,865 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:00,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:00,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:00,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:00,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:00,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:00,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:00,868 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:00,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:00,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:00,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:00,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:00,870 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:00,870 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:00,870 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:00,870 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:00,870 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:00,870 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:00,870 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:54:00,870 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:54:00,870 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:00,870 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:00,871 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:54:00,871 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:54:00,871 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:00,871 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:00,871 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:00,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:00,871 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:54:00,871 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:00,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:00,873 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=389 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/.tmp/B/91d4def1df8a407ca3c4852a1115a1d4 2023-04-25 02:54:00,971 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:54:00,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:54:00,973 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:54:00,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:54:00,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:54:00,975 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:54:00,975 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:00,975 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:54:00,975 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:54:00,975 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:00,976 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:00,977 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:00,977 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:00,977 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:00,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:00,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:00,978 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:00,978 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:54:00,978 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:00,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:54:00,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:00,979 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:54:00,979 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:54:00,979 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:54:00,979 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:54:00,979 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:00,979 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:54:00,979 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:54:00,979 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:54:00,980 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. 2023-04-25 02:54:00,980 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. started... 2023-04-25 02:54:00,980 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. as already flushing 2023-04-25 02:54:00,980 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. 2023-04-25 02:54:00,980 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:54:00,980 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:54:00,980 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:00,980 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:54:00,980 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:54:00,981 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:00,981 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:00,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:00,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:00,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:00,982 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:00,982 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:54:00,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:00,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:00,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:00,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:00,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:00,984 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:54:00,984 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@59145931[Count = 0] remaining members to acquire global barrier 2023-04-25 02:54:00,984 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:54:00,984 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:00,985 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:00,986 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:00,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:00,986 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:54:00,986 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:54:00,986 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:54:00,986 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:00,986 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:54:00,987 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:54:00,987 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:00,987 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:00,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:00,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:00,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:00,987 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:54:00,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:00,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:00,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:00,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:00,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:00,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:00,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:00,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:54:00,991 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:54:00,991 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:54:00,991 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:54:00,991 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:54:00,991 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:54:00,991 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:54:00,995 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:00,995 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:00,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:00,995 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:00,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:00,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:00,995 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:00,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:00,995 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:00,995 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:00,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:00,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:00,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:01,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:01,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:01,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:01,001 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:01,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:01,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:01,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:01,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:01,004 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:01,004 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:01,004 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:01,004 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:01,004 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:01,004 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:01,004 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:01,004 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:01,004 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:54:01,005 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:54:01,005 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:54:01,005 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:54:01,005 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:54:01,005 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:01,005 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:01,005 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:01,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:01,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:01,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:01,047 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:01,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43806 deadline: 1682391301045, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:01,056 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:01,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 209 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43822 deadline: 1682391301052, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:01,061 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:01,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43864 deadline: 1682391301057, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:01,105 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:54:01,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:54:01,108 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:54:01,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:54:01,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:54:01,110 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:54:01,110 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:01,110 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:54:01,110 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:54:01,110 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:01,110 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:01,113 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:01,113 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:01,114 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:01,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:01,114 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:01,114 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:54:01,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:01,114 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:01,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:54:01,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:01,115 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:54:01,115 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:54:01,115 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:54:01,115 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:54:01,115 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:01,115 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:54:01,116 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:54:01,116 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:54:01,116 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. 2023-04-25 02:54:01,116 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. started... 2023-04-25 02:54:01,116 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. as already flushing 2023-04-25 02:54:01,116 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. 2023-04-25 02:54:01,116 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:54:01,116 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:54:01,116 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:01,116 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:54:01,116 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:54:01,118 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:01,118 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:01,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:01,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:01,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:01,118 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:01,118 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:54:01,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:01,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:01,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:01,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:01,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:01,121 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:54:01,121 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@411f2519[Count = 0] remaining members to acquire global barrier 2023-04-25 02:54:01,121 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:54:01,121 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:01,122 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:01,122 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:01,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:01,123 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:54:01,123 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:54:01,123 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:01,123 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:54:01,123 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:54:01,124 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:54:01,124 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:01,124 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:01,124 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:54:01,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:01,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:01,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:01,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:01,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:01,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:01,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:01,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:01,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:01,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:01,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:54:01,127 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:54:01,127 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:54:01,127 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:54:01,127 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:54:01,127 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:54:01,127 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:54:01,130 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:01,130 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:01,130 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:01,130 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:01,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:01,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:01,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:01,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:01,130 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:01,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:01,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:01,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:01,132 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:01,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:01,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:01,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:01,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:01,134 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:01,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:01,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:01,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:01,136 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:01,137 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:01,137 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:01,136 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:01,137 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:01,137 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:01,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:01,137 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:01,137 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:01,137 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:01,137 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:01,137 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:01,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:01,138 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:01,138 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:54:01,138 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:54:01,138 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:54:01,138 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:54:01,238 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:54:01,239 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:54:01,240 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:54:01,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:54:01,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:54:01,242 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:54:01,242 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:01,242 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:54:01,242 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:54:01,243 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:01,243 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:01,247 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:01,247 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:01,247 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:01,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:01,247 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:01,248 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:54:01,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:01,248 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:01,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:54:01,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:01,248 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:54:01,249 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:54:01,249 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:54:01,249 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:54:01,249 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:01,249 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:54:01,249 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:54:01,249 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:54:01,249 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. 2023-04-25 02:54:01,249 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. started... 2023-04-25 02:54:01,249 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. as already flushing 2023-04-25 02:54:01,249 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. 2023-04-25 02:54:01,249 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:54:01,249 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:54:01,249 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:01,249 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:54:01,249 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:54:01,252 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:01,252 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:01,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:01,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:01,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:01,253 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:01,253 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:54:01,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:01,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:01,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:01,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:01,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:01,254 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:54:01,254 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@27de13f2[Count = 0] remaining members to acquire global barrier 2023-04-25 02:54:01,254 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:54:01,254 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:01,255 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:01,255 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:01,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:01,256 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:01,256 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:54:01,256 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:54:01,256 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:54:01,256 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:54:01,257 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:54:01,257 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:01,257 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:01,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:01,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:01,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:01,257 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:54:01,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:01,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:01,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:01,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:01,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:01,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:01,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:01,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:54:01,260 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:54:01,260 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:54:01,260 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:54:01,260 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:54:01,260 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:54:01,260 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:54:01,262 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:01,262 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:01,262 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:01,262 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:01,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:01,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:01,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:01,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:01,263 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:01,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:01,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:01,263 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:01,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:01,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:01,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:01,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:01,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:01,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:01,266 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:01,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:01,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:01,270 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:01,270 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:01,271 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:01,271 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:01,271 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:01,271 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:54:01,271 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:54:01,270 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:01,271 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:01,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:01,271 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:01,271 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:01,271 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:54:01,271 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:01,271 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:54:01,271 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:01,271 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:01,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:01,296 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=389 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/.tmp/C/1901c6c565834030a2105b84c0cd4297 2023-04-25 02:54:01,302 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/.tmp/A/fb6b7f329f11485b8c4e77d0daf433e0 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/A/fb6b7f329f11485b8c4e77d0daf433e0 2023-04-25 02:54:01,308 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/A/fb6b7f329f11485b8c4e77d0daf433e0, entries=150, sequenceid=389, filesize=11.9 K 2023-04-25 02:54:01,309 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/.tmp/B/91d4def1df8a407ca3c4852a1115a1d4 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/B/91d4def1df8a407ca3c4852a1115a1d4 2023-04-25 02:54:01,313 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/B/91d4def1df8a407ca3c4852a1115a1d4, entries=150, sequenceid=389, filesize=11.9 K 2023-04-25 02:54:01,314 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/.tmp/C/1901c6c565834030a2105b84c0cd4297 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/C/1901c6c565834030a2105b84c0cd4297 2023-04-25 02:54:01,319 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/C/1901c6c565834030a2105b84c0cd4297, entries=150, sequenceid=389, filesize=11.9 K 2023-04-25 02:54:01,320 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for 047cf722b8f6bff032073f994134764a in 920ms, sequenceid=389, compaction requested=true 2023-04-25 02:54:01,320 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 047cf722b8f6bff032073f994134764a: 2023-04-25 02:54:01,320 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 02:54:01,320 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-25 02:54:01,320 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-25 02:54:01,320 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 02:54:01,320 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 02:54:01,322 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 46914 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-25 02:54:01,322 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HStore(1912): 047cf722b8f6bff032073f994134764a/A is initiating minor compaction (all files) 2023-04-25 02:54:01,322 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 46914 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-25 02:54:01,322 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 047cf722b8f6bff032073f994134764a/A in TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. 2023-04-25 02:54:01,322 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HStore(1912): 047cf722b8f6bff032073f994134764a/B is initiating minor compaction (all files) 2023-04-25 02:54:01,322 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 047cf722b8f6bff032073f994134764a/B in TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. 2023-04-25 02:54:01,322 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/A/d8c8b27ce5b4407b83635cde50552811, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/A/b9669c3bf81348e8a2f28215cedb5d65, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/A/111876c02cfd48da805b4ee7cfc7579e, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/A/fb6b7f329f11485b8c4e77d0daf433e0] into tmpdir=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/.tmp, totalSize=45.8 K 2023-04-25 02:54:01,322 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/B/a1cc178c0a8a4cc1ab0b069edcea4299, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/B/47603300674344efab19e82c9c31815e, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/B/4bef52e5bba34900a2691e99883fd41b, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/B/91d4def1df8a407ca3c4852a1115a1d4] into tmpdir=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/.tmp, totalSize=45.8 K 2023-04-25 02:54:01,323 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting d8c8b27ce5b4407b83635cde50552811, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=340, earliestPutTs=1682391238134 2023-04-25 02:54:01,323 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting a1cc178c0a8a4cc1ab0b069edcea4299, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=340, earliestPutTs=1682391238134 2023-04-25 02:54:01,323 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting b9669c3bf81348e8a2f28215cedb5d65, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=352, earliestPutTs=1682391238488 2023-04-25 02:54:01,323 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting 47603300674344efab19e82c9c31815e, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=352, earliestPutTs=1682391238488 2023-04-25 02:54:01,324 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting 111876c02cfd48da805b4ee7cfc7579e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=375, earliestPutTs=1682391239636 2023-04-25 02:54:01,324 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting 4bef52e5bba34900a2691e99883fd41b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=375, earliestPutTs=1682391239636 2023-04-25 02:54:01,324 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting fb6b7f329f11485b8c4e77d0daf433e0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=389, earliestPutTs=1682391240285 2023-04-25 02:54:01,325 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting 91d4def1df8a407ca3c4852a1115a1d4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=389, earliestPutTs=1682391240285 2023-04-25 02:54:01,338 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] throttle.PressureAwareThroughputController(145): 047cf722b8f6bff032073f994134764a#B#compaction#67 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-25 02:54:01,338 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] throttle.PressureAwareThroughputController(145): 047cf722b8f6bff032073f994134764a#A#compaction#66 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-25 02:54:01,350 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/.tmp/B/3fad1af68b2f4a6e9f3225bb66d93611 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/B/3fad1af68b2f4a6e9f3225bb66d93611 2023-04-25 02:54:01,360 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 047cf722b8f6bff032073f994134764a/B of 047cf722b8f6bff032073f994134764a into 3fad1af68b2f4a6e9f3225bb66d93611(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 02:54:01,360 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 047cf722b8f6bff032073f994134764a: 2023-04-25 02:54:01,360 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a., storeName=047cf722b8f6bff032073f994134764a/B, priority=12, startTime=1682391241320; duration=0sec 2023-04-25 02:54:01,360 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 02:54:01,360 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-25 02:54:01,362 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 46914 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-25 02:54:01,362 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HStore(1912): 047cf722b8f6bff032073f994134764a/C is initiating minor compaction (all files) 2023-04-25 02:54:01,362 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 047cf722b8f6bff032073f994134764a/C in TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. 2023-04-25 02:54:01,362 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/C/1abffb9b26b44eaabb092b01c9237253, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/C/3642c7d23d994d8faf26b213d870e59e, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/C/e55d362edac649419466f512c3babd31, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/C/1901c6c565834030a2105b84c0cd4297] into tmpdir=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/.tmp, totalSize=45.8 K 2023-04-25 02:54:01,365 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting 1abffb9b26b44eaabb092b01c9237253, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=340, earliestPutTs=1682391238134 2023-04-25 02:54:01,366 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting 3642c7d23d994d8faf26b213d870e59e, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=352, earliestPutTs=1682391238488 2023-04-25 02:54:01,366 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting e55d362edac649419466f512c3babd31, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=375, earliestPutTs=1682391239636 2023-04-25 02:54:01,367 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting 1901c6c565834030a2105b84c0cd4297, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=389, earliestPutTs=1682391240285 2023-04-25 02:54:01,371 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:54:01,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:54:01,374 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:54:01,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:54:01,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:54:01,376 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:54:01,376 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:01,377 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:54:01,377 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:54:01,377 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:01,377 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:01,378 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] throttle.PressureAwareThroughputController(145): 047cf722b8f6bff032073f994134764a#C#compaction#68 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-25 02:54:01,379 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:01,379 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:01,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:01,379 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:01,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:01,380 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:01,380 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:54:01,380 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:01,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:54:01,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:01,381 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:54:01,384 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:54:01,384 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:54:01,384 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:54:01,384 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:01,384 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:54:01,385 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:54:01,385 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:54:01,385 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. 2023-04-25 02:54:01,385 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. started... 2023-04-25 02:54:01,385 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 047cf722b8f6bff032073f994134764a 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-25 02:54:01,386 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 047cf722b8f6bff032073f994134764a, store=A 2023-04-25 02:54:01,386 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:54:01,386 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 047cf722b8f6bff032073f994134764a, store=B 2023-04-25 02:54:01,386 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:54:01,386 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 047cf722b8f6bff032073f994134764a, store=C 2023-04-25 02:54:01,386 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:54:01,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(9158): Flush requested on 047cf722b8f6bff032073f994134764a 2023-04-25 02:54:01,554 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. as already flushing 2023-04-25 02:54:01,574 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:01,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43864 deadline: 1682391301570, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:01,577 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:01,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43822 deadline: 1682391301573, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:01,577 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:01,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 204 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43806 deadline: 1682391301574, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:01,677 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:01,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43864 deadline: 1682391301676, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:01,679 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:01,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 216 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43822 deadline: 1682391301678, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:01,680 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:01,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 206 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43806 deadline: 1682391301678, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:01,759 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/.tmp/A/b534ffe6c44e4642b9fdaf9e9a74e9f7 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/A/b534ffe6c44e4642b9fdaf9e9a74e9f7 2023-04-25 02:54:01,766 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 047cf722b8f6bff032073f994134764a/A of 047cf722b8f6bff032073f994134764a into b534ffe6c44e4642b9fdaf9e9a74e9f7(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 02:54:01,766 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 047cf722b8f6bff032073f994134764a: 2023-04-25 02:54:01,766 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a., storeName=047cf722b8f6bff032073f994134764a/A, priority=12, startTime=1682391241320; duration=0sec 2023-04-25 02:54:01,767 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 02:54:01,794 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=408 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/.tmp/A/f6c0e2d2a57e4f5798d49995d55c3cfa 2023-04-25 02:54:01,794 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/.tmp/C/b7643b512c3e4a94bb5bd1f4c92585d4 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/C/b7643b512c3e4a94bb5bd1f4c92585d4 2023-04-25 02:54:01,801 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 047cf722b8f6bff032073f994134764a/C of 047cf722b8f6bff032073f994134764a into b7643b512c3e4a94bb5bd1f4c92585d4(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 02:54:01,802 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 047cf722b8f6bff032073f994134764a: 2023-04-25 02:54:01,802 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a., storeName=047cf722b8f6bff032073f994134764a/C, priority=12, startTime=1682391241320; duration=0sec 2023-04-25 02:54:01,802 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 02:54:01,810 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=408 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/.tmp/B/a37210aee2574cdc96419668eeacb4cd 2023-04-25 02:54:01,826 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=408 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/.tmp/C/3033ce877f38460cb29dbbcb0d37e52b 2023-04-25 02:54:01,832 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/.tmp/A/f6c0e2d2a57e4f5798d49995d55c3cfa as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/A/f6c0e2d2a57e4f5798d49995d55c3cfa 2023-04-25 02:54:01,839 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/A/f6c0e2d2a57e4f5798d49995d55c3cfa, entries=150, sequenceid=408, filesize=11.9 K 2023-04-25 02:54:01,840 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/.tmp/B/a37210aee2574cdc96419668eeacb4cd as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/B/a37210aee2574cdc96419668eeacb4cd 2023-04-25 02:54:01,848 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/B/a37210aee2574cdc96419668eeacb4cd, entries=150, sequenceid=408, filesize=11.9 K 2023-04-25 02:54:01,848 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/.tmp/C/3033ce877f38460cb29dbbcb0d37e52b as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/C/3033ce877f38460cb29dbbcb0d37e52b 2023-04-25 02:54:01,856 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/C/3033ce877f38460cb29dbbcb0d37e52b, entries=150, sequenceid=408, filesize=11.9 K 2023-04-25 02:54:01,857 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for 047cf722b8f6bff032073f994134764a in 472ms, sequenceid=408, compaction requested=false 2023-04-25 02:54:01,858 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 047cf722b8f6bff032073f994134764a: 2023-04-25 02:54:01,858 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. 2023-04-25 02:54:01,858 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:54:01,858 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:54:01,858 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:01,858 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:54:01,858 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:54:01,860 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:01,860 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:01,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:01,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:01,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:01,860 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:01,860 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:54:01,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:01,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:01,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:01,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:01,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:01,862 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:54:01,862 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@543fd7ee[Count = 0] remaining members to acquire global barrier 2023-04-25 02:54:01,862 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:54:01,862 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:01,863 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:01,863 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:01,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:01,863 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:54:01,864 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:54:01,864 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:54:01,864 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:01,864 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:54:01,867 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:01,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:01,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:01,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:01,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:01,868 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:54:01,868 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:01,868 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:54:01,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:01,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:01,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:01,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:01,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:01,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:01,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:54:01,870 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:54:01,870 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:54:01,871 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:54:01,871 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:54:01,871 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:54:01,871 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:54:01,872 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:01,872 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:01,873 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:01,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:01,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:01,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:01,873 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:01,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:01,873 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:01,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:01,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:01,873 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:01,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:01,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:01,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:01,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:01,875 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:01,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:01,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:01,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:01,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:01,878 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:01,878 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:01,878 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:01,878 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:01,878 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:01,878 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:54:01,878 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:01,878 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:54:01,878 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:01,878 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:01,879 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:01,878 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:01,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:01,878 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:01,878 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:54:01,879 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:54:01,879 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:54:01,880 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:01,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:01,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(9158): Flush requested on 047cf722b8f6bff032073f994134764a 2023-04-25 02:54:01,882 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 047cf722b8f6bff032073f994134764a 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-25 02:54:01,884 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 047cf722b8f6bff032073f994134764a, store=A 2023-04-25 02:54:01,884 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:54:01,884 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 047cf722b8f6bff032073f994134764a, store=B 2023-04-25 02:54:01,884 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:54:01,884 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 047cf722b8f6bff032073f994134764a, store=C 2023-04-25 02:54:01,884 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:54:01,903 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=427 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/.tmp/A/7199ac36da0f4ccf840778e07fb29ac2 2023-04-25 02:54:01,907 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:01,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 223 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43822 deadline: 1682391301904, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:01,907 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:01,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 213 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43806 deadline: 1682391301904, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:01,908 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:01,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 204 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43864 deadline: 1682391301904, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:01,980 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:54:01,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:54:01,984 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:54:01,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:54:01,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:54:01,985 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:54:01,985 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:01,985 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:54:01,985 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:54:01,986 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:01,986 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:01,987 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:01,987 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:01,987 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:01,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:01,987 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:01,987 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:54:01,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:01,988 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:01,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:54:01,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:01,988 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:54:01,989 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:54:01,989 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:54:01,990 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:54:01,990 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:01,990 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:54:01,992 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:54:01,992 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:54:01,996 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. 2023-04-25 02:54:01,996 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. started... 2023-04-25 02:54:01,996 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. as already flushing 2023-04-25 02:54:01,996 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. 2023-04-25 02:54:01,996 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:54:01,996 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:54:01,996 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:01,996 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:54:01,996 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:54:01,999 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:01,999 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:01,999 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:54:02,000 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:02,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:02,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:02,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:02,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:02,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:02,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:02,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:02,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:02,002 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:54:02,002 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2f19d189[Count = 0] remaining members to acquire global barrier 2023-04-25 02:54:02,002 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:54:02,002 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:02,003 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:02,003 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:02,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:02,003 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:54:02,004 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:54:02,004 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:02,004 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:54:02,004 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:54:02,005 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:54:02,005 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:02,005 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:54:02,005 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:02,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:02,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:02,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:02,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:02,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:02,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:02,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:02,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:02,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:02,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:02,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:54:02,009 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:54:02,009 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:54:02,009 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:54:02,009 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:54:02,009 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:54:02,009 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:54:02,010 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:02,010 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:02,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:02,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:02,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:02,010 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:02,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:02,010 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:02,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:02,011 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:02,011 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:02,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:02,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:02,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:02,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:02,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:02,012 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:02,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:02,012 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:02,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 225 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43822 deadline: 1682391302008, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:02,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:02,013 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:02,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 215 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43806 deadline: 1682391302008, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:02,013 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:02,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:02,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 206 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43864 deadline: 1682391302009, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:02,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:02,016 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:02,016 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:02,016 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:02,016 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:02,016 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:02,016 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:02,016 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:02,016 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:02,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:02,016 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:02,016 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:54:02,016 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:54:02,016 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:02,016 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:02,016 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:54:02,017 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:54:02,017 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:02,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:02,017 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:54:02,117 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:54:02,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:54:02,119 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:54:02,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:54:02,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:54:02,120 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:54:02,120 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:02,120 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:54:02,120 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:54:02,120 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:02,121 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:02,122 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:02,122 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:02,122 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:02,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:02,122 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:02,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:02,122 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:54:02,123 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:02,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:54:02,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:02,123 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:54:02,123 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:54:02,123 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:54:02,123 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:54:02,123 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:02,124 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:54:02,124 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:54:02,124 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:54:02,124 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. 2023-04-25 02:54:02,124 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. started... 2023-04-25 02:54:02,124 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. as already flushing 2023-04-25 02:54:02,124 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. 2023-04-25 02:54:02,124 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:54:02,124 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:54:02,124 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:02,124 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:54:02,124 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:54:02,126 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:02,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:02,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:02,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:02,126 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:02,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:02,126 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:02,126 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:54:02,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:02,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:02,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:02,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:02,127 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:54:02,127 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@17961e9e[Count = 0] remaining members to acquire global barrier 2023-04-25 02:54:02,127 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:54:02,128 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:02,133 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:02,133 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:02,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:02,133 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:54:02,133 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:54:02,133 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:54:02,133 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:02,133 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:54:02,134 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:02,134 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:54:02,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:02,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:02,134 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:02,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:02,135 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:54:02,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:02,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:02,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:02,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:02,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:02,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:02,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:02,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:54:02,137 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:54:02,137 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:54:02,137 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:54:02,137 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:54:02,137 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:54:02,137 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:54:02,139 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:02,139 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:02,139 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:02,139 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:02,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:02,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:02,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:02,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:02,139 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:02,139 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:02,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:02,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:02,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:02,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:02,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:02,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:02,141 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:02,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:02,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:02,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:02,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:02,145 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:02,145 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:02,145 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:02,145 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:02,145 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:02,145 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:54:02,145 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:54:02,145 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:54:02,145 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:02,145 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:02,145 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:02,146 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:02,146 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:02,146 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:54:02,145 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:02,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:02,146 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:54:02,146 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:02,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:02,214 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:02,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43822 deadline: 1682391302214, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:02,215 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:02,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 217 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43806 deadline: 1682391302214, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:02,215 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:02,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 208 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43864 deadline: 1682391302214, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:02,246 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:54:02,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:54:02,249 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:54:02,250 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:54:02,250 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:54:02,250 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:54:02,250 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:02,250 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:54:02,250 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:54:02,251 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:02,251 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:02,253 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:02,253 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:02,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:02,253 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:02,253 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:02,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:02,253 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:54:02,254 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:02,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:54:02,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:02,254 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:54:02,254 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:54:02,255 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:54:02,255 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:54:02,255 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:02,255 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:54:02,255 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:54:02,255 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:54:02,255 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. 2023-04-25 02:54:02,255 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. started... 2023-04-25 02:54:02,255 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. as already flushing 2023-04-25 02:54:02,255 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. 2023-04-25 02:54:02,255 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:54:02,255 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:54:02,255 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:02,255 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:54:02,255 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:54:02,257 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:02,257 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:02,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:02,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:02,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:02,257 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:02,257 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:54:02,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:02,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:02,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:02,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:02,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:02,263 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:54:02,263 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4080de1d[Count = 0] remaining members to acquire global barrier 2023-04-25 02:54:02,263 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:54:02,263 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:02,265 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:02,265 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:02,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:02,265 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:54:02,265 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:54:02,265 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:54:02,265 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:02,265 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:54:02,268 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:02,268 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:54:02,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:02,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:02,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:02,268 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:02,268 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:54:02,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:02,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:02,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:02,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:02,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:02,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:02,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:02,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:54:02,272 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:54:02,272 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:54:02,272 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:54:02,272 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:54:02,272 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:54:02,272 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:54:02,275 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:02,275 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:02,275 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:02,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:02,275 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:02,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:02,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:02,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:02,276 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:02,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:02,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:02,276 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:02,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:02,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:02,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:02,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:02,277 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:02,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:02,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:02,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:02,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:02,279 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:02,280 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:02,280 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:02,280 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:02,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:02,280 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:02,280 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:02,280 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:54:02,280 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:54:02,280 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:02,280 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:02,280 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:02,280 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:02,280 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:02,280 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:54:02,280 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:02,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:02,280 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:54:02,280 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:54:02,332 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=427 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/.tmp/B/0ca4c656b11543a2863f3dfe91dc3564 2023-04-25 02:54:02,381 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:54:02,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:54:02,383 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:54:02,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:54:02,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:54:02,384 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:54:02,384 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:02,384 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:54:02,384 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:54:02,385 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:02,385 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:02,386 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:02,386 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:02,386 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:02,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:02,387 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:02,387 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:54:02,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:02,387 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:02,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:54:02,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:02,388 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:54:02,388 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:54:02,388 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:54:02,389 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:54:02,389 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:02,389 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:54:02,389 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:54:02,389 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:54:02,389 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. 2023-04-25 02:54:02,389 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. started... 2023-04-25 02:54:02,389 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. as already flushing 2023-04-25 02:54:02,389 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. 2023-04-25 02:54:02,389 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:54:02,389 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:54:02,389 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:02,389 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:54:02,389 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:54:02,392 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:02,392 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:02,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:02,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:02,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:02,392 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:02,392 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:54:02,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:02,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:02,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:02,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:02,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:02,393 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:54:02,393 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6aaaf144[Count = 0] remaining members to acquire global barrier 2023-04-25 02:54:02,393 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:54:02,393 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:02,394 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:02,395 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:02,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:02,395 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:54:02,395 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:54:02,395 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:54:02,395 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:02,395 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:54:02,396 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:02,396 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:54:02,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:02,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:02,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:02,396 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:02,396 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:54:02,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:02,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:02,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:02,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:02,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:02,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:02,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:02,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:54:02,400 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:54:02,400 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:54:02,400 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:54:02,400 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:54:02,400 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:54:02,400 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:54:02,402 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:02,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:02,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:02,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:02,404 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:02,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:02,404 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:02,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:02,404 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:02,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:02,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:02,405 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:02,406 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:02,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:02,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:02,406 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:02,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:02,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:02,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:02,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:02,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:02,409 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:02,409 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:02,409 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:02,409 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:02,409 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:02,409 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:02,409 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:54:02,409 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:54:02,409 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:02,410 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:02,410 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:02,409 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:02,409 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:02,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:02,410 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:54:02,410 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:54:02,410 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:02,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:02,411 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:54:02,510 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:54:02,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:54:02,513 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:54:02,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:54:02,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:54:02,515 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:54:02,515 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:02,515 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:54:02,516 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:54:02,516 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:02,516 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:02,518 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:02,518 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:02,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 229 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43822 deadline: 1682391302517, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:02,518 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:02,518 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:02,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:02,518 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:02,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43864 deadline: 1682391302517, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:02,519 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:02,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 219 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43806 deadline: 1682391302518, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:02,520 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:02,520 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:54:02,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:02,521 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:02,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:54:02,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:02,522 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:54:02,522 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:54:02,522 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:54:02,522 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:54:02,522 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:02,522 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:54:02,524 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:54:02,524 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:54:02,524 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. 2023-04-25 02:54:02,524 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. started... 2023-04-25 02:54:02,524 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. as already flushing 2023-04-25 02:54:02,524 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. 2023-04-25 02:54:02,524 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:54:02,524 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:54:02,524 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:02,524 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:54:02,524 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:54:02,525 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:02,525 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:02,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:02,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:02,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:02,526 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:02,526 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:54:02,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:02,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:02,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:02,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:02,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:02,528 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:54:02,528 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@b5ee001[Count = 0] remaining members to acquire global barrier 2023-04-25 02:54:02,528 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:54:02,528 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:02,529 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:02,529 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:02,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:02,529 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:54:02,530 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:02,530 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:54:02,530 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:54:02,530 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:54:02,535 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:02,535 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:54:02,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:02,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:02,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:02,535 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:02,535 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:54:02,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:02,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:02,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:02,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:02,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:02,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:02,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:02,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:54:02,538 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:54:02,538 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:54:02,538 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:54:02,538 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:54:02,538 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:54:02,538 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:54:02,540 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:02,540 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:02,540 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:02,540 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:02,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:02,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:02,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:02,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:02,540 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:02,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:02,541 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:02,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:02,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:02,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:02,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:02,542 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:02,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:02,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:02,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:02,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:02,544 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:02,544 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:02,544 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:02,544 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:02,545 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:02,545 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:02,545 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:02,545 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:02,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:02,545 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:02,545 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:54:02,545 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:54:02,545 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:02,545 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:54:02,545 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:02,545 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:54:02,545 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:02,545 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:54:02,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:02,645 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:54:02,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:54:02,648 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:54:02,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:54:02,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:54:02,649 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:54:02,649 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:02,649 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:54:02,649 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:54:02,649 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:02,650 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:02,652 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:02,652 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:02,652 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:02,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:02,652 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:02,652 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:54:02,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:02,653 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:02,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:54:02,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:02,653 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:54:02,654 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:54:02,654 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:54:02,654 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:54:02,654 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:02,654 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:54:02,654 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:54:02,654 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:54:02,654 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. 2023-04-25 02:54:02,654 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. started... 2023-04-25 02:54:02,654 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. as already flushing 2023-04-25 02:54:02,654 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. 2023-04-25 02:54:02,654 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:54:02,654 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:54:02,654 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:02,654 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:54:02,655 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:54:02,656 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:02,656 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:02,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:02,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:02,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:02,656 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:02,656 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:54:02,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:02,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:02,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:02,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:02,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:02,658 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:54:02,658 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2c9b7d1a[Count = 0] remaining members to acquire global barrier 2023-04-25 02:54:02,658 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:54:02,658 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:02,660 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:02,660 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:02,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:02,660 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:54:02,660 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:54:02,660 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:54:02,660 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:02,660 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:54:02,662 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:02,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:02,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:02,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:02,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:02,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:02,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:02,663 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:54:02,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:02,664 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:02,664 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:54:02,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:02,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:02,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:02,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:54:02,665 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:54:02,665 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:54:02,665 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:54:02,665 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:54:02,665 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:54:02,665 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:54:02,668 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:02,668 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:02,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:02,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:02,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:02,668 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:02,668 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:02,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:02,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:02,669 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:02,669 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:02,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:02,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:02,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:02,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:02,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:02,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:02,670 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:02,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:02,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:02,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:02,674 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:02,674 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:02,674 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:02,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:02,674 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:02,674 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:02,674 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:02,674 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:02,674 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:02,674 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:02,675 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:02,675 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:54:02,675 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:54:02,675 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:54:02,675 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:54:02,675 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:02,675 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:54:02,676 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:02,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:02,745 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=427 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/.tmp/C/4dec8146b8ed4e4281712b403b42dc94 2023-04-25 02:54:02,757 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/.tmp/A/7199ac36da0f4ccf840778e07fb29ac2 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/A/7199ac36da0f4ccf840778e07fb29ac2 2023-04-25 02:54:02,763 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/A/7199ac36da0f4ccf840778e07fb29ac2, entries=300, sequenceid=427, filesize=19.0 K 2023-04-25 02:54:02,764 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/.tmp/B/0ca4c656b11543a2863f3dfe91dc3564 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/B/0ca4c656b11543a2863f3dfe91dc3564 2023-04-25 02:54:02,773 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/B/0ca4c656b11543a2863f3dfe91dc3564, entries=150, sequenceid=427, filesize=11.9 K 2023-04-25 02:54:02,775 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/.tmp/C/4dec8146b8ed4e4281712b403b42dc94 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/C/4dec8146b8ed4e4281712b403b42dc94 2023-04-25 02:54:02,776 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:54:02,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:54:02,778 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:54:02,783 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/C/4dec8146b8ed4e4281712b403b42dc94, entries=150, sequenceid=427, filesize=11.9 K 2023-04-25 02:54:02,785 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=87.22 KB/89310 for 047cf722b8f6bff032073f994134764a in 903ms, sequenceid=427, compaction requested=true 2023-04-25 02:54:02,785 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 047cf722b8f6bff032073f994134764a: 2023-04-25 02:54:02,785 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 02:54:02,785 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 02:54:02,785 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-25 02:54:02,785 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-25 02:54:02,785 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-25 02:54:02,787 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 44628 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-25 02:54:02,788 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HStore(1912): 047cf722b8f6bff032073f994134764a/A is initiating minor compaction (all files) 2023-04-25 02:54:02,788 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 047cf722b8f6bff032073f994134764a/A in TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. 2023-04-25 02:54:02,788 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/A/b534ffe6c44e4642b9fdaf9e9a74e9f7, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/A/f6c0e2d2a57e4f5798d49995d55c3cfa, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/A/7199ac36da0f4ccf840778e07fb29ac2] into tmpdir=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/.tmp, totalSize=43.6 K 2023-04-25 02:54:02,789 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37308 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-25 02:54:02,789 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HStore(1912): 047cf722b8f6bff032073f994134764a/B is initiating minor compaction (all files) 2023-04-25 02:54:02,789 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 047cf722b8f6bff032073f994134764a/B in TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. 2023-04-25 02:54:02,789 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/B/3fad1af68b2f4a6e9f3225bb66d93611, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/B/a37210aee2574cdc96419668eeacb4cd, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/B/0ca4c656b11543a2863f3dfe91dc3564] into tmpdir=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/.tmp, totalSize=36.4 K 2023-04-25 02:54:02,790 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting 3fad1af68b2f4a6e9f3225bb66d93611, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=389, earliestPutTs=1682391240285 2023-04-25 02:54:02,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:54:02,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:54:02,790 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:54:02,790 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:02,790 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:54:02,790 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:54:02,790 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting b534ffe6c44e4642b9fdaf9e9a74e9f7, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=389, earliestPutTs=1682391240285 2023-04-25 02:54:02,791 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting a37210aee2574cdc96419668eeacb4cd, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=408, earliestPutTs=1682391240433 2023-04-25 02:54:02,791 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting f6c0e2d2a57e4f5798d49995d55c3cfa, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=408, earliestPutTs=1682391240433 2023-04-25 02:54:02,791 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting 0ca4c656b11543a2863f3dfe91dc3564, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=427, earliestPutTs=1682391241573 2023-04-25 02:54:02,792 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:02,792 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:02,793 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting 7199ac36da0f4ccf840778e07fb29ac2, keycount=300, bloomtype=ROW, size=19.0 K, encoding=NONE, compression=NONE, seqNum=427, earliestPutTs=1682391241567 2023-04-25 02:54:02,794 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:02,794 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:02,795 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:02,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:02,795 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:02,795 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:54:02,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:02,796 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:02,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:54:02,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:02,797 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:54:02,804 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:54:02,804 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:54:02,804 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:54:02,804 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:02,804 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:54:02,817 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] throttle.PressureAwareThroughputController(145): 047cf722b8f6bff032073f994134764a#B#compaction#75 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-25 02:54:02,818 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:54:02,818 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:54:02,818 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. 2023-04-25 02:54:02,818 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. started... 2023-04-25 02:54:02,819 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 047cf722b8f6bff032073f994134764a 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-25 02:54:02,819 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 047cf722b8f6bff032073f994134764a, store=A 2023-04-25 02:54:02,819 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:54:02,819 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 047cf722b8f6bff032073f994134764a, store=B 2023-04-25 02:54:02,819 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:54:02,819 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 047cf722b8f6bff032073f994134764a, store=C 2023-04-25 02:54:02,819 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:54:02,824 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] throttle.PressureAwareThroughputController(145): 047cf722b8f6bff032073f994134764a#A#compaction#76 average throughput is 1.64 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-25 02:54:02,870 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/.tmp/B/0eb0045209a848c597838fa367daf7fb as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/B/0eb0045209a848c597838fa367daf7fb 2023-04-25 02:54:02,879 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 047cf722b8f6bff032073f994134764a/B of 047cf722b8f6bff032073f994134764a into 0eb0045209a848c597838fa367daf7fb(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 02:54:02,880 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 047cf722b8f6bff032073f994134764a: 2023-04-25 02:54:02,880 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a., storeName=047cf722b8f6bff032073f994134764a/B, priority=13, startTime=1682391242785; duration=0sec 2023-04-25 02:54:02,880 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 02:54:02,880 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-25 02:54:02,882 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/.tmp/A/b82eeac6460e490a81536e523bc7f7cb as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/A/b82eeac6460e490a81536e523bc7f7cb 2023-04-25 02:54:02,883 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37308 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-25 02:54:02,883 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=443 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/.tmp/A/35b31049895446e18763ff8ed25b9c0c 2023-04-25 02:54:02,883 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HStore(1912): 047cf722b8f6bff032073f994134764a/C is initiating minor compaction (all files) 2023-04-25 02:54:02,884 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 047cf722b8f6bff032073f994134764a/C in TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. 2023-04-25 02:54:02,884 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/C/b7643b512c3e4a94bb5bd1f4c92585d4, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/C/3033ce877f38460cb29dbbcb0d37e52b, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/C/4dec8146b8ed4e4281712b403b42dc94] into tmpdir=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/.tmp, totalSize=36.4 K 2023-04-25 02:54:02,885 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting b7643b512c3e4a94bb5bd1f4c92585d4, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=389, earliestPutTs=1682391240285 2023-04-25 02:54:02,886 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting 3033ce877f38460cb29dbbcb0d37e52b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=408, earliestPutTs=1682391240433 2023-04-25 02:54:02,887 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting 4dec8146b8ed4e4281712b403b42dc94, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=427, earliestPutTs=1682391241573 2023-04-25 02:54:02,891 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 047cf722b8f6bff032073f994134764a/A of 047cf722b8f6bff032073f994134764a into b82eeac6460e490a81536e523bc7f7cb(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 02:54:02,891 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 047cf722b8f6bff032073f994134764a: 2023-04-25 02:54:02,891 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a., storeName=047cf722b8f6bff032073f994134764a/A, priority=13, startTime=1682391242785; duration=0sec 2023-04-25 02:54:02,891 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 02:54:02,905 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] throttle.PressureAwareThroughputController(145): 047cf722b8f6bff032073f994134764a#C#compaction#79 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-25 02:54:03,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(9158): Flush requested on 047cf722b8f6bff032073f994134764a 2023-04-25 02:54:03,022 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. as already flushing 2023-04-25 02:54:03,048 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:03,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 236 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43822 deadline: 1682391303040, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:03,051 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:03,051 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 216 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43864 deadline: 1682391303049, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:03,051 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:03,051 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 226 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43806 deadline: 1682391303049, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:03,151 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:03,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 238 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43822 deadline: 1682391303150, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:03,154 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:03,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 218 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43864 deadline: 1682391303152, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:03,154 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:03,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 228 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43806 deadline: 1682391303152, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:03,315 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=443 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/.tmp/B/d5f65a9d85874b2183f333ef174df201 2023-04-25 02:54:03,329 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/.tmp/C/7803152558ce417380a17a1703d2ea8c as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/C/7803152558ce417380a17a1703d2ea8c 2023-04-25 02:54:03,335 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=443 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/.tmp/C/8f429db4a2344354b7bcc28efd4ed505 2023-04-25 02:54:03,336 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 047cf722b8f6bff032073f994134764a/C of 047cf722b8f6bff032073f994134764a into 7803152558ce417380a17a1703d2ea8c(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 02:54:03,336 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 047cf722b8f6bff032073f994134764a: 2023-04-25 02:54:03,336 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a., storeName=047cf722b8f6bff032073f994134764a/C, priority=13, startTime=1682391242785; duration=0sec 2023-04-25 02:54:03,337 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 02:54:03,344 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/.tmp/A/35b31049895446e18763ff8ed25b9c0c as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/A/35b31049895446e18763ff8ed25b9c0c 2023-04-25 02:54:03,350 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/A/35b31049895446e18763ff8ed25b9c0c, entries=150, sequenceid=443, filesize=11.9 K 2023-04-25 02:54:03,351 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/.tmp/B/d5f65a9d85874b2183f333ef174df201 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/B/d5f65a9d85874b2183f333ef174df201 2023-04-25 02:54:03,356 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:03,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 240 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43822 deadline: 1682391303353, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:03,356 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:03,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 230 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43806 deadline: 1682391303355, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:03,356 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:03,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 220 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43864 deadline: 1682391303355, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:03,365 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/B/d5f65a9d85874b2183f333ef174df201, entries=150, sequenceid=443, filesize=11.9 K 2023-04-25 02:54:03,366 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/.tmp/C/8f429db4a2344354b7bcc28efd4ed505 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/C/8f429db4a2344354b7bcc28efd4ed505 2023-04-25 02:54:03,376 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/C/8f429db4a2344354b7bcc28efd4ed505, entries=150, sequenceid=443, filesize=11.9 K 2023-04-25 02:54:03,377 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=93.93 KB/96180 for 047cf722b8f6bff032073f994134764a in 558ms, sequenceid=443, compaction requested=false 2023-04-25 02:54:03,377 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 047cf722b8f6bff032073f994134764a: 2023-04-25 02:54:03,377 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. 2023-04-25 02:54:03,377 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:54:03,377 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:54:03,377 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:03,377 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:54:03,377 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:54:03,379 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:03,379 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:03,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:03,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:03,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:03,379 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:03,379 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:54:03,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:03,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:03,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:03,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:03,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:03,381 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:54:03,381 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6aad3ea2[Count = 0] remaining members to acquire global barrier 2023-04-25 02:54:03,381 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:54:03,381 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:03,384 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:03,384 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:03,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:03,384 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:54:03,384 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:54:03,384 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:54:03,384 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:03,384 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:54:03,386 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:03,386 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:54:03,386 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:03,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:03,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:03,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:03,386 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:54:03,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:03,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:03,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:03,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:03,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:03,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:03,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:03,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:54:03,389 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:54:03,389 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:54:03,389 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:54:03,389 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:54:03,389 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:54:03,389 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:54:03,391 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:03,391 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:03,391 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:03,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:03,391 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:03,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:03,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:03,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:03,391 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:03,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:03,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:03,392 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:03,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:03,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:03,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:03,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:03,393 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:03,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:03,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:03,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:03,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:03,395 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:03,395 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:03,395 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:03,396 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:03,396 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:03,396 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:03,396 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:03,396 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:03,396 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:03,396 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:03,396 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:54:03,396 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:54:03,396 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:03,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:03,396 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:54:03,396 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:54:03,397 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:03,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:03,497 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:54:03,497 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:54:03,499 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:54:03,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:54:03,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:54:03,501 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:54:03,501 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:03,501 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:54:03,501 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:54:03,501 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:03,501 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:03,504 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:03,504 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:03,504 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:03,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:03,504 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:03,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:03,504 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:54:03,505 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:03,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:54:03,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:03,505 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:54:03,505 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:54:03,505 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:54:03,505 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:54:03,506 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:03,506 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:54:03,506 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:54:03,506 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:54:03,506 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. 2023-04-25 02:54:03,506 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. started... 2023-04-25 02:54:03,506 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 047cf722b8f6bff032073f994134764a 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-25 02:54:03,506 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 047cf722b8f6bff032073f994134764a, store=A 2023-04-25 02:54:03,506 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:54:03,506 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 047cf722b8f6bff032073f994134764a, store=B 2023-04-25 02:54:03,506 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:54:03,506 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 047cf722b8f6bff032073f994134764a, store=C 2023-04-25 02:54:03,507 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:54:03,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(9158): Flush requested on 047cf722b8f6bff032073f994134764a 2023-04-25 02:54:03,661 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. as already flushing 2023-04-25 02:54:03,688 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:03,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 236 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43806 deadline: 1682391303679, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:03,691 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:03,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 246 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43822 deadline: 1682391303688, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:03,691 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:03,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43864 deadline: 1682391303688, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:03,790 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:03,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 238 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43806 deadline: 1682391303789, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:03,793 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:03,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 248 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43822 deadline: 1682391303792, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:03,794 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:03,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 229 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43864 deadline: 1682391303792, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:03,916 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=463 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/.tmp/A/13c7ee10b66047fba65cdb991362baa3 2023-04-25 02:54:03,936 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=463 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/.tmp/B/fbb8327919db4297b17076413d331c59 2023-04-25 02:54:03,952 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=463 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/.tmp/C/0bfa89b62d98492f9f018bd347e321d9 2023-04-25 02:54:03,957 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/.tmp/A/13c7ee10b66047fba65cdb991362baa3 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/A/13c7ee10b66047fba65cdb991362baa3 2023-04-25 02:54:03,964 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/A/13c7ee10b66047fba65cdb991362baa3, entries=150, sequenceid=463, filesize=11.9 K 2023-04-25 02:54:03,965 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/.tmp/B/fbb8327919db4297b17076413d331c59 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/B/fbb8327919db4297b17076413d331c59 2023-04-25 02:54:03,974 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/B/fbb8327919db4297b17076413d331c59, entries=150, sequenceid=463, filesize=11.9 K 2023-04-25 02:54:03,975 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/.tmp/C/0bfa89b62d98492f9f018bd347e321d9 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/C/0bfa89b62d98492f9f018bd347e321d9 2023-04-25 02:54:03,982 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/C/0bfa89b62d98492f9f018bd347e321d9, entries=150, sequenceid=463, filesize=11.9 K 2023-04-25 02:54:03,983 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=87.22 KB/89310 for 047cf722b8f6bff032073f994134764a in 477ms, sequenceid=463, compaction requested=true 2023-04-25 02:54:03,984 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 047cf722b8f6bff032073f994134764a: 2023-04-25 02:54:03,984 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. 2023-04-25 02:54:03,984 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:54:03,984 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:54:03,984 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:03,984 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:54:03,984 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:54:03,986 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:03,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:03,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:03,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:03,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:03,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:03,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:03,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:03,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:03,988 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:54:03,988 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5730c262[Count = 0] remaining members to acquire global barrier 2023-04-25 02:54:03,988 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:54:03,988 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:03,988 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:03,992 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:03,992 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:54:03,994 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:03,994 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:54:03,994 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:03,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:03,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:03,994 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:54:03,994 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:54:03,995 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:54:03,996 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:03,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:03,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:03,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:03,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:03,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:03,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:03,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:03,998 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:54:03,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:03,998 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:03,998 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:54:03,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:04,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(9158): Flush requested on 047cf722b8f6bff032073f994134764a 2023-04-25 02:54:04,001 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 047cf722b8f6bff032073f994134764a 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-25 02:54:04,003 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 047cf722b8f6bff032073f994134764a, store=A 2023-04-25 02:54:04,003 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:54:04,003 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 047cf722b8f6bff032073f994134764a, store=B 2023-04-25 02:54:04,003 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:54:04,003 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 047cf722b8f6bff032073f994134764a, store=C 2023-04-25 02:54:04,003 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:54:04,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:04,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:54:04,009 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:54:04,009 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:54:04,010 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:54:04,010 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:54:04,010 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:54:04,010 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:54:04,014 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:04,015 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:04,015 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:04,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:04,015 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:04,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:04,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:04,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:04,017 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:04,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:04,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:04,018 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:04,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:04,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:04,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:04,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:04,020 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:04,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:04,020 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=482 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/.tmp/A/b78b1140da834861b3c8b919bf8c02d5 2023-04-25 02:54:04,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:04,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:04,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:04,029 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:04,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 255 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43822 deadline: 1682391304027, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:04,029 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:04,030 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:04,029 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:54:04,030 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:04,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:04,030 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:54:04,029 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:04,030 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:04,030 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:04,030 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:04,030 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:04,030 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:04,030 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:04,030 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:04,030 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:54:04,031 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:54:04,031 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:54:04,032 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:04,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:04,034 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:04,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 245 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43806 deadline: 1682391304027, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:04,034 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:04,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 236 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43864 deadline: 1682391304028, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:04,036 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=482 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/.tmp/B/8772566f82b64e9aadf73038efc1207f 2023-04-25 02:54:04,131 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:54:04,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:54:04,133 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:04,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 257 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43822 deadline: 1682391304131, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:04,133 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:54:04,134 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:54:04,134 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:54:04,134 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:54:04,134 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:04,134 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:54:04,134 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:54:04,135 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:04,135 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:04,136 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:04,136 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:04,136 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:04,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:04,136 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:04,136 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:54:04,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:04,137 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:04,137 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:04,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:54:04,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:04,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 247 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43806 deadline: 1682391304135, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:04,137 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:54:04,137 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:54:04,137 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:54:04,138 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:54:04,138 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:04,138 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:54:04,138 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:54:04,138 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:54:04,138 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. 2023-04-25 02:54:04,138 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. started... 2023-04-25 02:54:04,138 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. as already flushing 2023-04-25 02:54:04,138 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. 2023-04-25 02:54:04,138 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:54:04,138 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:54:04,138 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:04,138 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:04,138 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:54:04,138 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:54:04,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 238 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43864 deadline: 1682391304136, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:04,140 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:04,140 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:04,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:04,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:04,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:04,140 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:04,140 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:54:04,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:04,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:04,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:04,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:04,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:04,142 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:54:04,142 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2b126e6d[Count = 0] remaining members to acquire global barrier 2023-04-25 02:54:04,142 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:54:04,142 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:04,143 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:04,143 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:04,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:04,143 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:54:04,143 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:54:04,143 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:54:04,143 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:04,143 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:54:04,146 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:04,146 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:54:04,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:04,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:04,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:04,146 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:04,146 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:54:04,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:04,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:04,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:04,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:04,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:04,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:04,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:04,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:54:04,148 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:54:04,148 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:54:04,148 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:54:04,148 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:54:04,148 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:54:04,148 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:54:04,150 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:04,150 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:04,150 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:04,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:04,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:04,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:04,150 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:04,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:04,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:04,150 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:04,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:04,151 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:04,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:04,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:04,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:04,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:04,152 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:04,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:04,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:04,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:04,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:04,154 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:04,154 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:04,154 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:04,154 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:04,154 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:04,154 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:04,154 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:04,155 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:54:04,155 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:54:04,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:04,154 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:04,155 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:04,155 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:04,155 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:04,155 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:54:04,155 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:54:04,155 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:54:04,155 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:04,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:04,255 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:54:04,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:54:04,257 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:54:04,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:54:04,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:54:04,259 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:54:04,259 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:04,259 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:54:04,259 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:54:04,259 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:04,259 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:04,261 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:04,261 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:04,261 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:04,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:04,261 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:04,261 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:54:04,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:04,261 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:04,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:54:04,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:04,262 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:54:04,262 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:54:04,262 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:54:04,262 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:54:04,262 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:04,262 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:54:04,262 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:54:04,262 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:54:04,262 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. 2023-04-25 02:54:04,262 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. started... 2023-04-25 02:54:04,263 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. as already flushing 2023-04-25 02:54:04,263 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. 2023-04-25 02:54:04,263 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:54:04,263 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:54:04,263 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:04,263 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:54:04,263 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:54:04,265 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:04,265 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:04,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:04,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:04,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:04,265 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:04,265 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:54:04,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:04,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:04,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:04,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:04,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:04,267 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:54:04,267 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5c96a5b8[Count = 0] remaining members to acquire global barrier 2023-04-25 02:54:04,267 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:54:04,267 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:04,268 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:04,268 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:04,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:04,268 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:54:04,268 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:54:04,268 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:54:04,268 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:04,268 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:54:04,270 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:04,270 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:54:04,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:04,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:04,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:04,270 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:04,270 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:54:04,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:04,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:04,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:04,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:04,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:04,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:04,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:04,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:54:04,272 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:54:04,272 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:54:04,273 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:54:04,273 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:54:04,273 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:54:04,273 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:54:04,274 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:04,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:04,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:04,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:04,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:04,276 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:04,276 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:04,276 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:04,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:04,276 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:04,276 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:04,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:04,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:04,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:04,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:04,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:04,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:04,278 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:04,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:04,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:04,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:04,281 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:04,281 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:04,281 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:04,281 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:04,281 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:54:04,281 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:54:04,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:04,281 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:04,281 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:04,281 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:04,281 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:04,281 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:54:04,281 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:54:04,281 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:04,281 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:04,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:04,282 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:04,282 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:04,339 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:04,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 259 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43822 deadline: 1682391304335, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:04,341 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:04,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 249 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43806 deadline: 1682391304338, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:04,342 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:04,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 240 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43864 deadline: 1682391304339, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:04,382 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:54:04,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:54:04,385 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:54:04,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:54:04,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:54:04,387 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:54:04,387 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:04,387 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:54:04,387 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:54:04,388 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:04,388 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:04,389 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:04,389 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:04,389 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:04,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:04,389 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:04,390 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:54:04,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:04,390 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:04,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:54:04,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:04,390 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:54:04,390 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:54:04,390 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:54:04,391 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:54:04,391 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:04,391 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:54:04,391 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:54:04,391 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:54:04,391 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. 2023-04-25 02:54:04,391 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. started... 2023-04-25 02:54:04,391 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. as already flushing 2023-04-25 02:54:04,391 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. 2023-04-25 02:54:04,391 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:54:04,391 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:54:04,391 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:04,391 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:54:04,391 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:54:04,393 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:04,393 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:04,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:04,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:04,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:04,393 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:04,393 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:54:04,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:04,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:04,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:04,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:04,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:04,394 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:54:04,394 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5fcdd8a6[Count = 0] remaining members to acquire global barrier 2023-04-25 02:54:04,394 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:54:04,394 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:04,395 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:04,395 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:04,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:04,396 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:54:04,396 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:54:04,396 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:04,396 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:54:04,396 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:54:04,398 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:54:04,398 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:04,398 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:04,398 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:54:04,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:04,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:04,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:04,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:04,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:04,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:04,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:04,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:04,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:04,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:04,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:54:04,400 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:54:04,400 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:54:04,401 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:54:04,401 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:54:04,401 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:54:04,401 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:54:04,402 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:04,402 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:04,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:04,402 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:04,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:04,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:04,402 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:04,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:04,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:04,403 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:04,403 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:04,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:04,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:04,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:04,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:04,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:04,404 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:04,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:04,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:04,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:04,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:04,414 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:04,414 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:04,414 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:54:04,414 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:54:04,414 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:04,414 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:04,414 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:04,414 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:04,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:04,414 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:04,415 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:54:04,415 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:04,415 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:54:04,415 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:04,415 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:54:04,415 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:04,415 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:04,415 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:04,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:04,471 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=482 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/.tmp/C/e3744fb9139c40fd8bd4b0c02b1c4e17 2023-04-25 02:54:04,481 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/.tmp/A/b78b1140da834861b3c8b919bf8c02d5 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/A/b78b1140da834861b3c8b919bf8c02d5 2023-04-25 02:54:04,489 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/A/b78b1140da834861b3c8b919bf8c02d5, entries=200, sequenceid=482, filesize=14.3 K 2023-04-25 02:54:04,490 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/.tmp/B/8772566f82b64e9aadf73038efc1207f as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/B/8772566f82b64e9aadf73038efc1207f 2023-04-25 02:54:04,495 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/B/8772566f82b64e9aadf73038efc1207f, entries=150, sequenceid=482, filesize=11.9 K 2023-04-25 02:54:04,496 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/.tmp/C/e3744fb9139c40fd8bd4b0c02b1c4e17 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/C/e3744fb9139c40fd8bd4b0c02b1c4e17 2023-04-25 02:54:04,502 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/C/e3744fb9139c40fd8bd4b0c02b1c4e17, entries=150, sequenceid=482, filesize=11.9 K 2023-04-25 02:54:04,503 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=80.51 KB/82440 for 047cf722b8f6bff032073f994134764a in 502ms, sequenceid=482, compaction requested=true 2023-04-25 02:54:04,503 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 047cf722b8f6bff032073f994134764a: 2023-04-25 02:54:04,503 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 02:54:04,504 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-25 02:54:04,504 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-25 02:54:04,504 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 02:54:04,504 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 02:54:04,505 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 52036 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-25 02:54:04,505 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HStore(1912): 047cf722b8f6bff032073f994134764a/A is initiating minor compaction (all files) 2023-04-25 02:54:04,506 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 047cf722b8f6bff032073f994134764a/A in TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. 2023-04-25 02:54:04,506 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49596 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-25 02:54:04,506 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HStore(1912): 047cf722b8f6bff032073f994134764a/B is initiating minor compaction (all files) 2023-04-25 02:54:04,506 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 047cf722b8f6bff032073f994134764a/B in TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. 2023-04-25 02:54:04,506 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/B/0eb0045209a848c597838fa367daf7fb, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/B/d5f65a9d85874b2183f333ef174df201, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/B/fbb8327919db4297b17076413d331c59, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/B/8772566f82b64e9aadf73038efc1207f] into tmpdir=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/.tmp, totalSize=48.4 K 2023-04-25 02:54:04,507 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/A/b82eeac6460e490a81536e523bc7f7cb, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/A/35b31049895446e18763ff8ed25b9c0c, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/A/13c7ee10b66047fba65cdb991362baa3, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/A/b78b1140da834861b3c8b919bf8c02d5] into tmpdir=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/.tmp, totalSize=50.8 K 2023-04-25 02:54:04,507 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting 0eb0045209a848c597838fa367daf7fb, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=427, earliestPutTs=1682391241573 2023-04-25 02:54:04,507 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting b82eeac6460e490a81536e523bc7f7cb, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=427, earliestPutTs=1682391241573 2023-04-25 02:54:04,507 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting d5f65a9d85874b2183f333ef174df201, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=443, earliestPutTs=1682391241899 2023-04-25 02:54:04,508 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting 35b31049895446e18763ff8ed25b9c0c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=443, earliestPutTs=1682391241899 2023-04-25 02:54:04,508 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting fbb8327919db4297b17076413d331c59, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=463, earliestPutTs=1682391243037 2023-04-25 02:54:04,508 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting 13c7ee10b66047fba65cdb991362baa3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=463, earliestPutTs=1682391243037 2023-04-25 02:54:04,509 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting 8772566f82b64e9aadf73038efc1207f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=482, earliestPutTs=1682391243679 2023-04-25 02:54:04,509 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting b78b1140da834861b3c8b919bf8c02d5, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=482, earliestPutTs=1682391243679 2023-04-25 02:54:04,515 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:54:04,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:54:04,521 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:54:04,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:54:04,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:54:04,523 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:54:04,523 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:04,523 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:54:04,523 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:54:04,523 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:04,523 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:04,523 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] throttle.PressureAwareThroughputController(145): 047cf722b8f6bff032073f994134764a#B#compaction#87 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-25 02:54:04,525 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:04,525 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:04,526 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:04,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:04,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:04,526 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:04,526 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:54:04,526 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:04,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:54:04,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:04,528 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:54:04,529 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:54:04,529 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:54:04,529 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:54:04,529 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:04,529 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:54:04,532 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:54:04,532 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:54:04,532 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. 2023-04-25 02:54:04,532 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. started... 2023-04-25 02:54:04,532 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 047cf722b8f6bff032073f994134764a 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-25 02:54:04,532 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 047cf722b8f6bff032073f994134764a, store=A 2023-04-25 02:54:04,532 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:54:04,532 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 047cf722b8f6bff032073f994134764a, store=B 2023-04-25 02:54:04,532 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:54:04,533 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 047cf722b8f6bff032073f994134764a, store=C 2023-04-25 02:54:04,533 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:54:04,533 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] throttle.PressureAwareThroughputController(145): 047cf722b8f6bff032073f994134764a#A#compaction#88 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-25 02:54:04,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(9158): Flush requested on 047cf722b8f6bff032073f994134764a 2023-04-25 02:54:04,648 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. as already flushing 2023-04-25 02:54:04,675 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:04,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 247 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43864 deadline: 1682391304668, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:04,680 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:04,680 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:04,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 256 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43806 deadline: 1682391304676, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:04,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 266 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43822 deadline: 1682391304675, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:04,778 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:04,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 249 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43864 deadline: 1682391304776, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:04,784 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:04,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 258 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43806 deadline: 1682391304782, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:04,785 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:04,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 268 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43822 deadline: 1682391304782, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:04,944 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=497 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/.tmp/A/ba1264621545429aa21212513b67a563 2023-04-25 02:54:04,945 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/.tmp/B/1b27355b70b74540a21c68cc714516f8 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/B/1b27355b70b74540a21c68cc714516f8 2023-04-25 02:54:04,951 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/.tmp/A/5c112e7d7e424bcabfe39eab8b99141b as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/A/5c112e7d7e424bcabfe39eab8b99141b 2023-04-25 02:54:04,959 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 047cf722b8f6bff032073f994134764a/B of 047cf722b8f6bff032073f994134764a into 1b27355b70b74540a21c68cc714516f8(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 02:54:04,959 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 047cf722b8f6bff032073f994134764a: 2023-04-25 02:54:04,959 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a., storeName=047cf722b8f6bff032073f994134764a/B, priority=12, startTime=1682391244504; duration=0sec 2023-04-25 02:54:04,960 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 02:54:04,960 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-25 02:54:04,962 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49596 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-25 02:54:04,962 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HStore(1912): 047cf722b8f6bff032073f994134764a/C is initiating minor compaction (all files) 2023-04-25 02:54:04,962 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 047cf722b8f6bff032073f994134764a/C in TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. 2023-04-25 02:54:04,963 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/C/7803152558ce417380a17a1703d2ea8c, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/C/8f429db4a2344354b7bcc28efd4ed505, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/C/0bfa89b62d98492f9f018bd347e321d9, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/C/e3744fb9139c40fd8bd4b0c02b1c4e17] into tmpdir=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/.tmp, totalSize=48.4 K 2023-04-25 02:54:04,964 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting 7803152558ce417380a17a1703d2ea8c, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=427, earliestPutTs=1682391241573 2023-04-25 02:54:04,964 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 047cf722b8f6bff032073f994134764a/A of 047cf722b8f6bff032073f994134764a into 5c112e7d7e424bcabfe39eab8b99141b(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 02:54:04,964 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 047cf722b8f6bff032073f994134764a: 2023-04-25 02:54:04,964 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a., storeName=047cf722b8f6bff032073f994134764a/A, priority=12, startTime=1682391244503; duration=0sec 2023-04-25 02:54:04,964 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 02:54:04,965 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting 8f429db4a2344354b7bcc28efd4ed505, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=443, earliestPutTs=1682391241899 2023-04-25 02:54:04,965 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting 0bfa89b62d98492f9f018bd347e321d9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=463, earliestPutTs=1682391243037 2023-04-25 02:54:04,966 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting e3744fb9139c40fd8bd4b0c02b1c4e17, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=482, earliestPutTs=1682391243679 2023-04-25 02:54:04,982 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:04,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 251 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43864 deadline: 1682391304980, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:04,987 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:04,987 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 270 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43822 deadline: 1682391304987, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:04,990 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] throttle.PressureAwareThroughputController(145): 047cf722b8f6bff032073f994134764a#C#compaction#91 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-25 02:54:04,990 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:04,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 260 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43806 deadline: 1682391304988, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:05,287 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:05,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 253 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43864 deadline: 1682391305285, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:05,291 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:05,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 272 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43822 deadline: 1682391305290, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:05,295 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:05,295 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 262 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43806 deadline: 1682391305294, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:05,387 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=497 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/.tmp/B/469e398a68764ec0bc5178e47806fa85 2023-04-25 02:54:05,406 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=497 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/.tmp/C/89eb8cf7c552478da0435e3e96a59353 2023-04-25 02:54:05,412 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/.tmp/A/ba1264621545429aa21212513b67a563 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/A/ba1264621545429aa21212513b67a563 2023-04-25 02:54:05,418 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/A/ba1264621545429aa21212513b67a563, entries=150, sequenceid=497, filesize=11.9 K 2023-04-25 02:54:05,424 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/.tmp/B/469e398a68764ec0bc5178e47806fa85 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/B/469e398a68764ec0bc5178e47806fa85 2023-04-25 02:54:05,430 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/B/469e398a68764ec0bc5178e47806fa85, entries=150, sequenceid=497, filesize=11.9 K 2023-04-25 02:54:05,431 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/.tmp/C/89eb8cf7c552478da0435e3e96a59353 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/C/89eb8cf7c552478da0435e3e96a59353 2023-04-25 02:54:05,431 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/.tmp/C/faa5dc9f820d49acb143842cce854e50 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/C/faa5dc9f820d49acb143842cce854e50 2023-04-25 02:54:05,436 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/C/89eb8cf7c552478da0435e3e96a59353, entries=150, sequenceid=497, filesize=11.9 K 2023-04-25 02:54:05,437 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=100.63 KB/103050 for 047cf722b8f6bff032073f994134764a in 905ms, sequenceid=497, compaction requested=false 2023-04-25 02:54:05,437 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 047cf722b8f6bff032073f994134764a: 2023-04-25 02:54:05,437 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. 2023-04-25 02:54:05,437 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:54:05,437 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:54:05,437 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:05,438 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 047cf722b8f6bff032073f994134764a/C of 047cf722b8f6bff032073f994134764a into faa5dc9f820d49acb143842cce854e50(size=12.9 K), total size for store is 24.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 02:54:05,438 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 047cf722b8f6bff032073f994134764a: 2023-04-25 02:54:05,438 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a., storeName=047cf722b8f6bff032073f994134764a/C, priority=12, startTime=1682391244504; duration=0sec 2023-04-25 02:54:05,438 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 02:54:05,440 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:54:05,440 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:54:05,441 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:05,441 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:05,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:05,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:05,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:05,442 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:05,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:05,442 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:54:05,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:05,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:05,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:05,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:05,443 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:54:05,443 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@ebf22ef[Count = 0] remaining members to acquire global barrier 2023-04-25 02:54:05,443 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:54:05,443 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:05,444 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:05,444 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:05,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:05,444 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:54:05,444 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:05,445 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:54:05,445 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:54:05,445 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:54:05,446 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:05,446 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:54:05,446 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:05,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:05,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:05,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:05,446 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:54:05,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:05,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:05,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:05,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:05,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:05,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:05,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:05,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:54:05,448 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:54:05,448 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:54:05,448 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:54:05,448 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:54:05,448 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:54:05,448 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:54:05,451 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:05,451 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:05,451 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:05,451 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:05,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:05,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:05,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:05,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:05,452 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:05,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:05,452 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:05,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:05,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:05,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:05,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:05,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:05,453 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:05,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:05,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:05,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:05,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:05,455 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:05,455 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:05,455 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:05,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:05,456 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:05,456 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:54:05,456 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:54:05,456 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:05,456 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:05,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:05,456 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:05,456 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:05,456 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:05,456 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:05,456 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:54:05,456 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:05,456 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:54:05,456 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:05,456 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:54:05,517 DEBUG [Thread-156] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0d24bbb7 to 127.0.0.1:62054 2023-04-25 02:54:05,517 DEBUG [Thread-156] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-25 02:54:05,518 DEBUG [Thread-152] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x49d71120 to 127.0.0.1:62054 2023-04-25 02:54:05,518 DEBUG [Thread-152] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-25 02:54:05,519 DEBUG [Thread-158] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x63871a9d to 127.0.0.1:62054 2023-04-25 02:54:05,519 DEBUG [Thread-158] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-25 02:54:05,521 DEBUG [Thread-154] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x59d6b6ea to 127.0.0.1:62054 2023-04-25 02:54:05,521 DEBUG [Thread-154] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-25 02:54:05,556 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:54:05,557 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:54:05,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(9158): Flush requested on 047cf722b8f6bff032073f994134764a 2023-04-25 02:54:05,793 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 047cf722b8f6bff032073f994134764a 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-25 02:54:05,794 DEBUG [Thread-141] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x748f869b to 127.0.0.1:62054 2023-04-25 02:54:05,794 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 047cf722b8f6bff032073f994134764a, store=A 2023-04-25 02:54:05,794 DEBUG [Thread-141] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-25 02:54:05,794 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:54:05,794 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 047cf722b8f6bff032073f994134764a, store=B 2023-04-25 02:54:05,794 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:54:05,794 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 047cf722b8f6bff032073f994134764a, store=C 2023-04-25 02:54:05,794 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:54:05,798 DEBUG [Thread-143] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x56223d42 to 127.0.0.1:62054 2023-04-25 02:54:05,798 DEBUG [Thread-143] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-25 02:54:05,798 DEBUG [Thread-149] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x02c57e1b to 127.0.0.1:62054 2023-04-25 02:54:05,798 DEBUG [Thread-149] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-25 02:54:06,204 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=519 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/.tmp/A/99b2d951846d4909a9ea50d0fc7c590e 2023-04-25 02:54:06,315 DEBUG [Thread-147] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2de4d1f6 to 127.0.0.1:62054 2023-04-25 02:54:06,315 DEBUG [Thread-147] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-25 02:54:06,363 DEBUG [Thread-145] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x30eb7fb2 to 127.0.0.1:62054 2023-04-25 02:54:06,363 DEBUG [Thread-145] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-25 02:54:06,363 INFO [Listener at localhost/41915] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-25 02:54:06,363 INFO [Listener at localhost/41915] hbase.AcidGuaranteesTestTool(394): wrote 125 2023-04-25 02:54:06,363 INFO [Listener at localhost/41915] hbase.AcidGuaranteesTestTool(394): wrote 134 2023-04-25 02:54:06,363 INFO [Listener at localhost/41915] hbase.AcidGuaranteesTestTool(394): wrote 15 2023-04-25 02:54:06,363 INFO [Listener at localhost/41915] hbase.AcidGuaranteesTestTool(394): wrote 18 2023-04-25 02:54:06,363 INFO [Listener at localhost/41915] hbase.AcidGuaranteesTestTool(394): wrote 134 2023-04-25 02:54:06,363 INFO [Listener at localhost/41915] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-25 02:54:06,363 INFO [Listener at localhost/41915] hbase.AcidGuaranteesTestTool(398): read 4203 2023-04-25 02:54:06,363 INFO [Listener at localhost/41915] hbase.AcidGuaranteesTestTool(398): read 4233 2023-04-25 02:54:06,363 INFO [Listener at localhost/41915] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-25 02:54:06,363 INFO [Listener at localhost/41915] hbase.AcidGuaranteesTestTool(402): scanned 1836 2023-04-25 02:54:06,363 INFO [Listener at localhost/41915] hbase.AcidGuaranteesTestTool(403): verified 5504 rows 2023-04-25 02:54:06,364 INFO [Listener at localhost/41915] hbase.AcidGuaranteesTestTool(402): scanned 1845 2023-04-25 02:54:06,364 INFO [Listener at localhost/41915] hbase.AcidGuaranteesTestTool(403): verified 5533 rows 2023-04-25 02:54:06,364 INFO [Listener at localhost/41915] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-25 02:54:06,364 DEBUG [Listener at localhost/41915] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x08c3f619 to 127.0.0.1:62054 2023-04-25 02:54:06,364 DEBUG [Listener at localhost/41915] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-25 02:54:06,369 INFO [Listener at localhost/41915] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-25 02:54:06,374 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-04-25 02:54:06,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure2.ProcedureExecutor(1029): Stored pid=12, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-25 02:54:06,383 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682391246383"}]},"ts":"1682391246383"} 2023-04-25 02:54:06,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-25 02:54:06,384 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-25 02:54:06,387 INFO [PEWorker-3] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-25 02:54:06,389 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=13, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=047cf722b8f6bff032073f994134764a, UNASSIGN}] 2023-04-25 02:54:06,390 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=13, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=047cf722b8f6bff032073f994134764a, UNASSIGN 2023-04-25 02:54:06,391 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=047cf722b8f6bff032073f994134764a, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:06,391 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682391246391"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682391246391"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682391246391"}]},"ts":"1682391246391"} 2023-04-25 02:54:06,392 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=14, ppid=13, state=RUNNABLE; CloseRegionProcedure 047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023}] 2023-04-25 02:54:06,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-25 02:54:06,551 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 047cf722b8f6bff032073f994134764a 2023-04-25 02:54:06,551 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 047cf722b8f6bff032073f994134764a, disabling compactions & flushes 2023-04-25 02:54:06,551 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1858): waiting for 0 compactions & cache flush to complete for region TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. 2023-04-25 02:54:06,615 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=519 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/.tmp/B/90c111166e924ee58f0ac11ead4d1045 2023-04-25 02:54:06,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-25 02:54:06,987 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-25 02:54:07,027 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=519 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/.tmp/C/3d1ad5643ae64cf5a74438dfa4c0a1f3 2023-04-25 02:54:07,032 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/.tmp/A/99b2d951846d4909a9ea50d0fc7c590e as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/A/99b2d951846d4909a9ea50d0fc7c590e 2023-04-25 02:54:07,037 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/A/99b2d951846d4909a9ea50d0fc7c590e, entries=150, sequenceid=519, filesize=11.9 K 2023-04-25 02:54:07,037 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/.tmp/B/90c111166e924ee58f0ac11ead4d1045 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/B/90c111166e924ee58f0ac11ead4d1045 2023-04-25 02:54:07,042 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/B/90c111166e924ee58f0ac11ead4d1045, entries=150, sequenceid=519, filesize=11.9 K 2023-04-25 02:54:07,043 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/.tmp/C/3d1ad5643ae64cf5a74438dfa4c0a1f3 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/C/3d1ad5643ae64cf5a74438dfa4c0a1f3 2023-04-25 02:54:07,048 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/C/3d1ad5643ae64cf5a74438dfa4c0a1f3, entries=150, sequenceid=519, filesize=11.9 K 2023-04-25 02:54:07,049 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=26.84 KB/27480 for 047cf722b8f6bff032073f994134764a in 1256ms, sequenceid=519, compaction requested=true 2023-04-25 02:54:07,049 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 047cf722b8f6bff032073f994134764a: 2023-04-25 02:54:07,049 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. 2023-04-25 02:54:07,049 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 02:54:07,049 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. 2023-04-25 02:54:07,049 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 02:54:07,049 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. after waiting 0 ms 2023-04-25 02:54:07,049 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 02:54:07,049 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. because compaction request was cancelled 2023-04-25 02:54:07,049 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. because compaction request was cancelled 2023-04-25 02:54:07,049 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. because compaction request was cancelled 2023-04-25 02:54:07,049 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. 2023-04-25 02:54:07,049 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 047cf722b8f6bff032073f994134764a 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-25 02:54:07,049 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 047cf722b8f6bff032073f994134764a, store=A 2023-04-25 02:54:07,049 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:54:07,049 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 047cf722b8f6bff032073f994134764a, store=B 2023-04-25 02:54:07,049 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:54:07,049 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 047cf722b8f6bff032073f994134764a, store=C 2023-04-25 02:54:07,050 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:54:07,458 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=526 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/.tmp/A/81013bc8bf3f4bfeb020b7fc88b93348 2023-04-25 02:54:07,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-25 02:54:07,870 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=526 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/.tmp/B/76b47e29c8574ef395dc9b52592c6850 2023-04-25 02:54:08,281 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=526 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/.tmp/C/ee9050058706404b9f12656672d2e5a7 2023-04-25 02:54:08,286 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/.tmp/A/81013bc8bf3f4bfeb020b7fc88b93348 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/A/81013bc8bf3f4bfeb020b7fc88b93348 2023-04-25 02:54:08,290 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/A/81013bc8bf3f4bfeb020b7fc88b93348, entries=150, sequenceid=526, filesize=11.9 K 2023-04-25 02:54:08,291 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/.tmp/B/76b47e29c8574ef395dc9b52592c6850 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/B/76b47e29c8574ef395dc9b52592c6850 2023-04-25 02:54:08,296 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/B/76b47e29c8574ef395dc9b52592c6850, entries=150, sequenceid=526, filesize=11.9 K 2023-04-25 02:54:08,297 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/.tmp/C/ee9050058706404b9f12656672d2e5a7 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/C/ee9050058706404b9f12656672d2e5a7 2023-04-25 02:54:08,301 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/C/ee9050058706404b9f12656672d2e5a7, entries=150, sequenceid=526, filesize=11.9 K 2023-04-25 02:54:08,302 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=0 B/0 for 047cf722b8f6bff032073f994134764a in 1253ms, sequenceid=526, compaction requested=true 2023-04-25 02:54:08,323 DEBUG [StoreCloser-TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/A/fb1f6146377f462f98c9d568f13113d1, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/A/8c319e53504e434e92f0783fec8e57a1, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/A/cc671b2ab2ae4be78ebbe8537bc3c01b, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/A/30024e1bd1f44345b0cf6d4b6d9c89b7, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/A/d0ee89d7f1d143a5a70d112d5ae6fb22, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/A/9db33aa5cd3840b2ae0a076d53bc8f0d, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/A/eaaa86ea137f4bd684bfed0cd21d311b, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/A/f8b1717c091841429271f9c3f14e32b5, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/A/a60c550bd51e4ae1877e85b1ebc0036a, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/A/92e055129a1b4b4e80bd8b072e04aa61, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/A/32c71762857c40e4b6ba25edc682c6c4, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/A/e219f9629b2c44f5985325caecc7cd97, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/A/4dab330a3a2e4a76b62b2e37126a74cf, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/A/2b90aa83e9e24e1898b7def9a4e87a69, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/A/66cc459630d54035a3bef400504e929c, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/A/f42105b99bed441c900da64221720246, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/A/09cc91e9cdd44c12bba2e7c3bebe89df, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/A/4d00fb6ac870456a8d76d669c1102d68, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/A/d8c8b27ce5b4407b83635cde50552811, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/A/b9669c3bf81348e8a2f28215cedb5d65, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/A/111876c02cfd48da805b4ee7cfc7579e, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/A/b534ffe6c44e4642b9fdaf9e9a74e9f7, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/A/fb6b7f329f11485b8c4e77d0daf433e0, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/A/f6c0e2d2a57e4f5798d49995d55c3cfa, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/A/7199ac36da0f4ccf840778e07fb29ac2, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/A/b82eeac6460e490a81536e523bc7f7cb, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/A/35b31049895446e18763ff8ed25b9c0c, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/A/13c7ee10b66047fba65cdb991362baa3, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/A/b78b1140da834861b3c8b919bf8c02d5] to archive 2023-04-25 02:54:08,324 DEBUG [StoreCloser-TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-25 02:54:08,330 DEBUG [StoreCloser-TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/A/fb1f6146377f462f98c9d568f13113d1 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/A/fb1f6146377f462f98c9d568f13113d1 2023-04-25 02:54:08,331 DEBUG [StoreCloser-TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/A/8c319e53504e434e92f0783fec8e57a1 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/A/8c319e53504e434e92f0783fec8e57a1 2023-04-25 02:54:08,333 DEBUG [StoreCloser-TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/A/cc671b2ab2ae4be78ebbe8537bc3c01b to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/A/cc671b2ab2ae4be78ebbe8537bc3c01b 2023-04-25 02:54:08,334 DEBUG [StoreCloser-TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/A/30024e1bd1f44345b0cf6d4b6d9c89b7 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/A/30024e1bd1f44345b0cf6d4b6d9c89b7 2023-04-25 02:54:08,335 DEBUG [StoreCloser-TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/A/d0ee89d7f1d143a5a70d112d5ae6fb22 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/A/d0ee89d7f1d143a5a70d112d5ae6fb22 2023-04-25 02:54:08,336 DEBUG [StoreCloser-TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/A/9db33aa5cd3840b2ae0a076d53bc8f0d to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/A/9db33aa5cd3840b2ae0a076d53bc8f0d 2023-04-25 02:54:08,337 DEBUG [StoreCloser-TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/A/eaaa86ea137f4bd684bfed0cd21d311b to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/A/eaaa86ea137f4bd684bfed0cd21d311b 2023-04-25 02:54:08,338 DEBUG [StoreCloser-TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/A/f8b1717c091841429271f9c3f14e32b5 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/A/f8b1717c091841429271f9c3f14e32b5 2023-04-25 02:54:08,339 DEBUG [StoreCloser-TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/A/a60c550bd51e4ae1877e85b1ebc0036a to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/A/a60c550bd51e4ae1877e85b1ebc0036a 2023-04-25 02:54:08,341 DEBUG [StoreCloser-TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/A/92e055129a1b4b4e80bd8b072e04aa61 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/A/92e055129a1b4b4e80bd8b072e04aa61 2023-04-25 02:54:08,342 DEBUG [StoreCloser-TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/A/32c71762857c40e4b6ba25edc682c6c4 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/A/32c71762857c40e4b6ba25edc682c6c4 2023-04-25 02:54:08,343 DEBUG [StoreCloser-TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/A/e219f9629b2c44f5985325caecc7cd97 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/A/e219f9629b2c44f5985325caecc7cd97 2023-04-25 02:54:08,344 DEBUG [StoreCloser-TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/A/4dab330a3a2e4a76b62b2e37126a74cf to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/A/4dab330a3a2e4a76b62b2e37126a74cf 2023-04-25 02:54:08,345 DEBUG [StoreCloser-TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/A/2b90aa83e9e24e1898b7def9a4e87a69 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/A/2b90aa83e9e24e1898b7def9a4e87a69 2023-04-25 02:54:08,346 DEBUG [StoreCloser-TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/A/66cc459630d54035a3bef400504e929c to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/A/66cc459630d54035a3bef400504e929c 2023-04-25 02:54:08,348 DEBUG [StoreCloser-TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/A/f42105b99bed441c900da64221720246 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/A/f42105b99bed441c900da64221720246 2023-04-25 02:54:08,349 DEBUG [StoreCloser-TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/A/09cc91e9cdd44c12bba2e7c3bebe89df to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/A/09cc91e9cdd44c12bba2e7c3bebe89df 2023-04-25 02:54:08,350 DEBUG [StoreCloser-TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/A/4d00fb6ac870456a8d76d669c1102d68 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/A/4d00fb6ac870456a8d76d669c1102d68 2023-04-25 02:54:08,352 DEBUG [StoreCloser-TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/A/d8c8b27ce5b4407b83635cde50552811 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/A/d8c8b27ce5b4407b83635cde50552811 2023-04-25 02:54:08,354 DEBUG [StoreCloser-TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/A/b9669c3bf81348e8a2f28215cedb5d65 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/A/b9669c3bf81348e8a2f28215cedb5d65 2023-04-25 02:54:08,355 DEBUG [StoreCloser-TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/A/111876c02cfd48da805b4ee7cfc7579e to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/A/111876c02cfd48da805b4ee7cfc7579e 2023-04-25 02:54:08,357 DEBUG [StoreCloser-TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/A/b534ffe6c44e4642b9fdaf9e9a74e9f7 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/A/b534ffe6c44e4642b9fdaf9e9a74e9f7 2023-04-25 02:54:08,360 DEBUG [StoreCloser-TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/A/fb6b7f329f11485b8c4e77d0daf433e0 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/A/fb6b7f329f11485b8c4e77d0daf433e0 2023-04-25 02:54:08,361 DEBUG [StoreCloser-TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/A/f6c0e2d2a57e4f5798d49995d55c3cfa to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/A/f6c0e2d2a57e4f5798d49995d55c3cfa 2023-04-25 02:54:08,362 DEBUG [StoreCloser-TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/A/7199ac36da0f4ccf840778e07fb29ac2 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/A/7199ac36da0f4ccf840778e07fb29ac2 2023-04-25 02:54:08,363 DEBUG [StoreCloser-TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/A/b82eeac6460e490a81536e523bc7f7cb to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/A/b82eeac6460e490a81536e523bc7f7cb 2023-04-25 02:54:08,364 DEBUG [StoreCloser-TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/A/35b31049895446e18763ff8ed25b9c0c to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/A/35b31049895446e18763ff8ed25b9c0c 2023-04-25 02:54:08,366 DEBUG [StoreCloser-TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/A/13c7ee10b66047fba65cdb991362baa3 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/A/13c7ee10b66047fba65cdb991362baa3 2023-04-25 02:54:08,370 DEBUG [StoreCloser-TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/A/b78b1140da834861b3c8b919bf8c02d5 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/A/b78b1140da834861b3c8b919bf8c02d5 2023-04-25 02:54:08,396 DEBUG [StoreCloser-TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/B/c6eb3e4c7d404fddb2a1a2a946cb4489, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/B/a480c509a8124c1e9216dcba81c7f1e3, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/B/80c60c631b7c454681537f934d1dd2b1, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/B/cbc5f7506be1404695aff07919df8ad1, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/B/df13aedd944e410185ba3fadc0fb7e83, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/B/01bd58c8af354be18d171a3022afad59, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/B/e956d8789b0f4cf497b5c380548a9a4b, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/B/2fe07082cf584ff5a43ecb56ea7a4271, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/B/27a973d536f244d2ba53f04fa4895ff7, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/B/06ce65c0646b4c498be7edc184a4c269, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/B/5400faad1fa64b0a9f4a926928c5c481, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/B/a43d2baa2ae7496c95a0e390d104b651, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/B/85cc0d029a104e25a26fce78c5275423, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/B/a09959b0235e46eca0ae302da0c1b63c, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/B/6b8cced2b09542a18ef2dacac211e4b3, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/B/f2c22b8fae3e473995c520ed366ba64d, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/B/d64320d988474929a0088055f27703f6, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/B/a1cc178c0a8a4cc1ab0b069edcea4299, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/B/148dc052a953472382a76365c9c783e2, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/B/47603300674344efab19e82c9c31815e, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/B/4bef52e5bba34900a2691e99883fd41b, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/B/3fad1af68b2f4a6e9f3225bb66d93611, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/B/91d4def1df8a407ca3c4852a1115a1d4, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/B/a37210aee2574cdc96419668eeacb4cd, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/B/0eb0045209a848c597838fa367daf7fb, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/B/0ca4c656b11543a2863f3dfe91dc3564, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/B/d5f65a9d85874b2183f333ef174df201, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/B/fbb8327919db4297b17076413d331c59, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/B/8772566f82b64e9aadf73038efc1207f] to archive 2023-04-25 02:54:08,397 DEBUG [StoreCloser-TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-25 02:54:08,398 DEBUG [StoreCloser-TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/B/c6eb3e4c7d404fddb2a1a2a946cb4489 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/B/c6eb3e4c7d404fddb2a1a2a946cb4489 2023-04-25 02:54:08,399 DEBUG [StoreCloser-TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/B/a480c509a8124c1e9216dcba81c7f1e3 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/B/a480c509a8124c1e9216dcba81c7f1e3 2023-04-25 02:54:08,400 DEBUG [StoreCloser-TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/B/80c60c631b7c454681537f934d1dd2b1 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/B/80c60c631b7c454681537f934d1dd2b1 2023-04-25 02:54:08,401 DEBUG [StoreCloser-TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/B/cbc5f7506be1404695aff07919df8ad1 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/B/cbc5f7506be1404695aff07919df8ad1 2023-04-25 02:54:08,403 DEBUG [StoreCloser-TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/B/df13aedd944e410185ba3fadc0fb7e83 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/B/df13aedd944e410185ba3fadc0fb7e83 2023-04-25 02:54:08,404 DEBUG [StoreCloser-TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/B/01bd58c8af354be18d171a3022afad59 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/B/01bd58c8af354be18d171a3022afad59 2023-04-25 02:54:08,405 DEBUG [StoreCloser-TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/B/e956d8789b0f4cf497b5c380548a9a4b to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/B/e956d8789b0f4cf497b5c380548a9a4b 2023-04-25 02:54:08,406 DEBUG [StoreCloser-TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/B/2fe07082cf584ff5a43ecb56ea7a4271 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/B/2fe07082cf584ff5a43ecb56ea7a4271 2023-04-25 02:54:08,407 DEBUG [StoreCloser-TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/B/27a973d536f244d2ba53f04fa4895ff7 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/B/27a973d536f244d2ba53f04fa4895ff7 2023-04-25 02:54:08,408 DEBUG [StoreCloser-TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/B/06ce65c0646b4c498be7edc184a4c269 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/B/06ce65c0646b4c498be7edc184a4c269 2023-04-25 02:54:08,409 DEBUG [StoreCloser-TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/B/5400faad1fa64b0a9f4a926928c5c481 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/B/5400faad1fa64b0a9f4a926928c5c481 2023-04-25 02:54:08,410 DEBUG [StoreCloser-TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/B/a43d2baa2ae7496c95a0e390d104b651 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/B/a43d2baa2ae7496c95a0e390d104b651 2023-04-25 02:54:08,411 DEBUG [StoreCloser-TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/B/85cc0d029a104e25a26fce78c5275423 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/B/85cc0d029a104e25a26fce78c5275423 2023-04-25 02:54:08,412 DEBUG [StoreCloser-TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/B/a09959b0235e46eca0ae302da0c1b63c to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/B/a09959b0235e46eca0ae302da0c1b63c 2023-04-25 02:54:08,413 DEBUG [StoreCloser-TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/B/6b8cced2b09542a18ef2dacac211e4b3 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/B/6b8cced2b09542a18ef2dacac211e4b3 2023-04-25 02:54:08,414 DEBUG [StoreCloser-TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/B/f2c22b8fae3e473995c520ed366ba64d to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/B/f2c22b8fae3e473995c520ed366ba64d 2023-04-25 02:54:08,415 DEBUG [StoreCloser-TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/B/d64320d988474929a0088055f27703f6 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/B/d64320d988474929a0088055f27703f6 2023-04-25 02:54:08,416 DEBUG [StoreCloser-TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/B/a1cc178c0a8a4cc1ab0b069edcea4299 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/B/a1cc178c0a8a4cc1ab0b069edcea4299 2023-04-25 02:54:08,417 DEBUG [StoreCloser-TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/B/148dc052a953472382a76365c9c783e2 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/B/148dc052a953472382a76365c9c783e2 2023-04-25 02:54:08,418 DEBUG [StoreCloser-TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/B/47603300674344efab19e82c9c31815e to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/B/47603300674344efab19e82c9c31815e 2023-04-25 02:54:08,419 DEBUG [StoreCloser-TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/B/4bef52e5bba34900a2691e99883fd41b to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/B/4bef52e5bba34900a2691e99883fd41b 2023-04-25 02:54:08,420 DEBUG [StoreCloser-TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/B/3fad1af68b2f4a6e9f3225bb66d93611 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/B/3fad1af68b2f4a6e9f3225bb66d93611 2023-04-25 02:54:08,421 DEBUG [StoreCloser-TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/B/91d4def1df8a407ca3c4852a1115a1d4 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/B/91d4def1df8a407ca3c4852a1115a1d4 2023-04-25 02:54:08,422 DEBUG [StoreCloser-TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/B/a37210aee2574cdc96419668eeacb4cd to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/B/a37210aee2574cdc96419668eeacb4cd 2023-04-25 02:54:08,423 DEBUG [StoreCloser-TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/B/0eb0045209a848c597838fa367daf7fb to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/B/0eb0045209a848c597838fa367daf7fb 2023-04-25 02:54:08,424 DEBUG [StoreCloser-TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/B/0ca4c656b11543a2863f3dfe91dc3564 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/B/0ca4c656b11543a2863f3dfe91dc3564 2023-04-25 02:54:08,425 DEBUG [StoreCloser-TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/B/d5f65a9d85874b2183f333ef174df201 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/B/d5f65a9d85874b2183f333ef174df201 2023-04-25 02:54:08,426 DEBUG [StoreCloser-TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/B/fbb8327919db4297b17076413d331c59 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/B/fbb8327919db4297b17076413d331c59 2023-04-25 02:54:08,427 DEBUG [StoreCloser-TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/B/8772566f82b64e9aadf73038efc1207f to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/B/8772566f82b64e9aadf73038efc1207f 2023-04-25 02:54:08,430 DEBUG [StoreCloser-TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/C/e1c283bc917340499cff80cee4e3ccef, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/C/cc68fe1357cb40e083f89f1f51013e29, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/C/d978d53d0452462cb593a4a9d336b81a, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/C/91aef605175a4841876534a48a41031a, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/C/79721a49470e41aab987a4934a3f715d, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/C/f22a298a041a411fa9e6163622766315, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/C/1ee24ddf8c5744828d3deb8cbbf45a91, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/C/5192eaba12f14abba2a9d04a287cc410, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/C/f305449eff62412ea77f94b8dd919930, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/C/c6473c9403cd427f8dd388cf662b9c8f, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/C/03f0784e7fb34b3c8345582879f1d6fe, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/C/e28cc196ac12477c920283f072ddc5e5, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/C/cf47055f6bfa4336bba1737336d5fcef, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/C/5827a2ca106c4a58b0e61b09103fe534, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/C/813081adb8a9484a895c1cbbbb341063, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/C/b887bfeb411c4be6ad60af38c202355c, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/C/5bb31b7ca4204cbdb08a05d2a9c69c3e, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/C/1abffb9b26b44eaabb092b01c9237253, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/C/d843da56122c4c96b30e528f6d37dbc8, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/C/3642c7d23d994d8faf26b213d870e59e, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/C/e55d362edac649419466f512c3babd31, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/C/b7643b512c3e4a94bb5bd1f4c92585d4, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/C/1901c6c565834030a2105b84c0cd4297, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/C/3033ce877f38460cb29dbbcb0d37e52b, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/C/7803152558ce417380a17a1703d2ea8c, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/C/4dec8146b8ed4e4281712b403b42dc94, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/C/8f429db4a2344354b7bcc28efd4ed505, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/C/0bfa89b62d98492f9f018bd347e321d9, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/C/e3744fb9139c40fd8bd4b0c02b1c4e17] to archive 2023-04-25 02:54:08,431 DEBUG [StoreCloser-TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-25 02:54:08,432 DEBUG [StoreCloser-TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/C/e1c283bc917340499cff80cee4e3ccef to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/C/e1c283bc917340499cff80cee4e3ccef 2023-04-25 02:54:08,434 DEBUG [StoreCloser-TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/C/cc68fe1357cb40e083f89f1f51013e29 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/C/cc68fe1357cb40e083f89f1f51013e29 2023-04-25 02:54:08,435 DEBUG [StoreCloser-TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/C/d978d53d0452462cb593a4a9d336b81a to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/C/d978d53d0452462cb593a4a9d336b81a 2023-04-25 02:54:08,436 DEBUG [StoreCloser-TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/C/91aef605175a4841876534a48a41031a to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/C/91aef605175a4841876534a48a41031a 2023-04-25 02:54:08,437 DEBUG [StoreCloser-TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/C/79721a49470e41aab987a4934a3f715d to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/C/79721a49470e41aab987a4934a3f715d 2023-04-25 02:54:08,439 DEBUG [StoreCloser-TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/C/f22a298a041a411fa9e6163622766315 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/C/f22a298a041a411fa9e6163622766315 2023-04-25 02:54:08,440 DEBUG [StoreCloser-TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/C/1ee24ddf8c5744828d3deb8cbbf45a91 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/C/1ee24ddf8c5744828d3deb8cbbf45a91 2023-04-25 02:54:08,441 DEBUG [StoreCloser-TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/C/5192eaba12f14abba2a9d04a287cc410 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/C/5192eaba12f14abba2a9d04a287cc410 2023-04-25 02:54:08,442 DEBUG [StoreCloser-TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/C/f305449eff62412ea77f94b8dd919930 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/C/f305449eff62412ea77f94b8dd919930 2023-04-25 02:54:08,443 DEBUG [StoreCloser-TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/C/c6473c9403cd427f8dd388cf662b9c8f to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/C/c6473c9403cd427f8dd388cf662b9c8f 2023-04-25 02:54:08,445 DEBUG [StoreCloser-TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/C/03f0784e7fb34b3c8345582879f1d6fe to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/C/03f0784e7fb34b3c8345582879f1d6fe 2023-04-25 02:54:08,446 DEBUG [StoreCloser-TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/C/e28cc196ac12477c920283f072ddc5e5 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/C/e28cc196ac12477c920283f072ddc5e5 2023-04-25 02:54:08,447 DEBUG [StoreCloser-TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/C/cf47055f6bfa4336bba1737336d5fcef to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/C/cf47055f6bfa4336bba1737336d5fcef 2023-04-25 02:54:08,448 DEBUG [StoreCloser-TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/C/5827a2ca106c4a58b0e61b09103fe534 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/C/5827a2ca106c4a58b0e61b09103fe534 2023-04-25 02:54:08,449 DEBUG [StoreCloser-TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/C/813081adb8a9484a895c1cbbbb341063 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/C/813081adb8a9484a895c1cbbbb341063 2023-04-25 02:54:08,450 DEBUG [StoreCloser-TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/C/b887bfeb411c4be6ad60af38c202355c to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/C/b887bfeb411c4be6ad60af38c202355c 2023-04-25 02:54:08,452 DEBUG [StoreCloser-TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/C/5bb31b7ca4204cbdb08a05d2a9c69c3e to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/C/5bb31b7ca4204cbdb08a05d2a9c69c3e 2023-04-25 02:54:08,453 DEBUG [StoreCloser-TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/C/1abffb9b26b44eaabb092b01c9237253 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/C/1abffb9b26b44eaabb092b01c9237253 2023-04-25 02:54:08,454 DEBUG [StoreCloser-TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/C/d843da56122c4c96b30e528f6d37dbc8 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/C/d843da56122c4c96b30e528f6d37dbc8 2023-04-25 02:54:08,456 DEBUG [StoreCloser-TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/C/3642c7d23d994d8faf26b213d870e59e to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/C/3642c7d23d994d8faf26b213d870e59e 2023-04-25 02:54:08,457 DEBUG [StoreCloser-TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/C/e55d362edac649419466f512c3babd31 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/C/e55d362edac649419466f512c3babd31 2023-04-25 02:54:08,458 DEBUG [StoreCloser-TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/C/b7643b512c3e4a94bb5bd1f4c92585d4 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/C/b7643b512c3e4a94bb5bd1f4c92585d4 2023-04-25 02:54:08,459 DEBUG [StoreCloser-TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/C/1901c6c565834030a2105b84c0cd4297 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/C/1901c6c565834030a2105b84c0cd4297 2023-04-25 02:54:08,460 DEBUG [StoreCloser-TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/C/3033ce877f38460cb29dbbcb0d37e52b to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/C/3033ce877f38460cb29dbbcb0d37e52b 2023-04-25 02:54:08,461 DEBUG [StoreCloser-TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/C/7803152558ce417380a17a1703d2ea8c to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/C/7803152558ce417380a17a1703d2ea8c 2023-04-25 02:54:08,463 DEBUG [StoreCloser-TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/C/4dec8146b8ed4e4281712b403b42dc94 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/C/4dec8146b8ed4e4281712b403b42dc94 2023-04-25 02:54:08,464 DEBUG [StoreCloser-TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/C/8f429db4a2344354b7bcc28efd4ed505 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/C/8f429db4a2344354b7bcc28efd4ed505 2023-04-25 02:54:08,465 DEBUG [StoreCloser-TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/C/0bfa89b62d98492f9f018bd347e321d9 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/C/0bfa89b62d98492f9f018bd347e321d9 2023-04-25 02:54:08,466 DEBUG [StoreCloser-TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/C/e3744fb9139c40fd8bd4b0c02b1c4e17 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/C/e3744fb9139c40fd8bd4b0c02b1c4e17 2023-04-25 02:54:08,472 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/recovered.edits/529.seqid, newMaxSeqId=529, maxSeqId=1 2023-04-25 02:54:08,473 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a. 2023-04-25 02:54:08,473 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 047cf722b8f6bff032073f994134764a: 2023-04-25 02:54:08,475 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 047cf722b8f6bff032073f994134764a 2023-04-25 02:54:08,476 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=047cf722b8f6bff032073f994134764a, regionState=CLOSED 2023-04-25 02:54:08,476 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682391248476"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682391248476"}]},"ts":"1682391248476"} 2023-04-25 02:54:08,479 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=14, resume processing ppid=13 2023-04-25 02:54:08,479 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=14, ppid=13, state=SUCCESS; CloseRegionProcedure 047cf722b8f6bff032073f994134764a, server=jenkins-hbase4.apache.org,46697,1682391220023 in 2.0850 sec 2023-04-25 02:54:08,480 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=13, resume processing ppid=12 2023-04-25 02:54:08,480 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=13, ppid=12, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=047cf722b8f6bff032073f994134764a, UNASSIGN in 2.0900 sec 2023-04-25 02:54:08,481 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682391248481"}]},"ts":"1682391248481"} 2023-04-25 02:54:08,482 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-25 02:54:08,483 INFO [PEWorker-3] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-25 02:54:08,485 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=12, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 2.1090 sec 2023-04-25 02:54:08,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-25 02:54:08,489 INFO [Listener at localhost/41915] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 12 completed 2023-04-25 02:54:08,493 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-04-25 02:54:08,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure2.ProcedureExecutor(1029): Stored pid=15, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-25 02:54:08,501 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=15, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-25 02:54:08,502 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=15, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-25 02:54:08,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-04-25 02:54:08,507 DEBUG [HFileArchiver-3] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/.tmp/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a 2023-04-25 02:54:08,510 DEBUG [HFileArchiver-3] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/.tmp/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/A, FileablePath, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/.tmp/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/B, FileablePath, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/.tmp/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/C, FileablePath, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/.tmp/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/recovered.edits] 2023-04-25 02:54:08,513 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/.tmp/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/A/5c112e7d7e424bcabfe39eab8b99141b to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/A/5c112e7d7e424bcabfe39eab8b99141b 2023-04-25 02:54:08,515 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/.tmp/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/A/81013bc8bf3f4bfeb020b7fc88b93348 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/A/81013bc8bf3f4bfeb020b7fc88b93348 2023-04-25 02:54:08,516 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/.tmp/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/A/99b2d951846d4909a9ea50d0fc7c590e to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/A/99b2d951846d4909a9ea50d0fc7c590e 2023-04-25 02:54:08,518 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/.tmp/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/A/ba1264621545429aa21212513b67a563 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/A/ba1264621545429aa21212513b67a563 2023-04-25 02:54:08,520 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/.tmp/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/B/1b27355b70b74540a21c68cc714516f8 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/B/1b27355b70b74540a21c68cc714516f8 2023-04-25 02:54:08,522 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/.tmp/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/B/469e398a68764ec0bc5178e47806fa85 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/B/469e398a68764ec0bc5178e47806fa85 2023-04-25 02:54:08,523 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/.tmp/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/B/76b47e29c8574ef395dc9b52592c6850 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/B/76b47e29c8574ef395dc9b52592c6850 2023-04-25 02:54:08,524 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/.tmp/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/B/90c111166e924ee58f0ac11ead4d1045 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/B/90c111166e924ee58f0ac11ead4d1045 2023-04-25 02:54:08,528 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/.tmp/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/C/3d1ad5643ae64cf5a74438dfa4c0a1f3 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/C/3d1ad5643ae64cf5a74438dfa4c0a1f3 2023-04-25 02:54:08,529 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/.tmp/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/C/89eb8cf7c552478da0435e3e96a59353 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/C/89eb8cf7c552478da0435e3e96a59353 2023-04-25 02:54:08,530 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/.tmp/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/C/ee9050058706404b9f12656672d2e5a7 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/C/ee9050058706404b9f12656672d2e5a7 2023-04-25 02:54:08,532 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/.tmp/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/C/faa5dc9f820d49acb143842cce854e50 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/C/faa5dc9f820d49acb143842cce854e50 2023-04-25 02:54:08,535 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/.tmp/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/recovered.edits/529.seqid to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a/recovered.edits/529.seqid 2023-04-25 02:54:08,536 DEBUG [HFileArchiver-3] backup.HFileArchiver(596): Deleted hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/.tmp/data/default/TestAcidGuarantees/047cf722b8f6bff032073f994134764a 2023-04-25 02:54:08,536 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-25 02:54:08,538 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=15, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-25 02:54:08,561 WARN [PEWorker-5] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-25 02:54:08,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-04-25 02:54:08,615 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-25 02:54:08,616 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=15, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-25 02:54:08,616 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-25 02:54:08,617 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1682391248616"}]},"ts":"9223372036854775807"} 2023-04-25 02:54:08,619 INFO [PEWorker-5] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-25 02:54:08,619 DEBUG [PEWorker-5] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 047cf722b8f6bff032073f994134764a, NAME => 'TestAcidGuarantees,,1682391224263.047cf722b8f6bff032073f994134764a.', STARTKEY => '', ENDKEY => ''}] 2023-04-25 02:54:08,619 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-25 02:54:08,619 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1682391248619"}]},"ts":"9223372036854775807"} 2023-04-25 02:54:08,621 INFO [PEWorker-5] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-25 02:54:08,624 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(130): Finished pid=15, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-25 02:54:08,625 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=15, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 130 msec 2023-04-25 02:54:08,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-04-25 02:54:08,806 INFO [Listener at localhost/41915] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 15 completed 2023-04-25 02:54:08,823 INFO [Listener at localhost/41915] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMixedAtomicity Thread=235 (was 206) Potentially hanging thread: (jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-599764414_15 at /127.0.0.1:38610 [Waiting for operation #4] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-3 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-8 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-13 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-599764414_15 at /127.0.0.1:38612 [Waiting for operation #9] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-2 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:883) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-599764414_15 at /127.0.0.1:38604 [Waiting for operation #6] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x14e0c1f2-shared-pool-16 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-12 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Timer for 'HBase' metrics system java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-10 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x14e0c1f2-shared-pool-17 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-1-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-7 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-4 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-599764414_15 at /127.0.0.1:38700 [Waiting for operation #3] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x14e0c1f2-shared-pool-18 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x14e0c1f2-shared-pool-19 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-9 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-6 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RSProcedureDispatcher-pool-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-5 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS:0;jenkins-hbase4:46697-shortCompactions-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.PriorityBlockingQueue.take(PriorityBlockingQueue.java:549) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-11 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) - Thread LEAK? -, OpenFileDescriptor=577 (was 470) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=560 (was 409) - SystemLoadAverage LEAK? -, ProcessCount=171 (was 170) - ProcessCount LEAK? -, AvailableMemoryMB=4486 (was 5108) 2023-04-25 02:54:08,835 INFO [Listener at localhost/41915] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMobMixedAtomicity Thread=235, OpenFileDescriptor=577, MaxFileDescriptor=60000, SystemLoadAverage=560, ProcessCount=170, AvailableMemoryMB=4485 2023-04-25 02:54:08,837 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-25 02:54:08,838 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-25 02:54:08,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure2.ProcedureExecutor(1029): Stored pid=16, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-25 02:54:08,841 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-25 02:54:08,841 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 16 2023-04-25 02:54:08,842 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-25 02:54:08,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-25 02:54:08,844 DEBUG [HFileArchiver-4] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/.tmp/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198 2023-04-25 02:54:08,844 DEBUG [HFileArchiver-4] backup.HFileArchiver(153): Directory hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/.tmp/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198 empty. 2023-04-25 02:54:08,845 DEBUG [HFileArchiver-4] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/.tmp/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198 2023-04-25 02:54:08,845 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-25 02:54:08,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-25 02:54:09,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-25 02:54:09,254 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-25 02:54:09,256 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 388bc1373cd34a10ba8be8ddc731e198, NAME => 'TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/.tmp 2023-04-25 02:54:09,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-25 02:54:09,662 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-25 02:54:09,662 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 388bc1373cd34a10ba8be8ddc731e198, disabling compactions & flushes 2023-04-25 02:54:09,662 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. 2023-04-25 02:54:09,662 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. 2023-04-25 02:54:09,662 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. after waiting 0 ms 2023-04-25 02:54:09,662 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. 2023-04-25 02:54:09,663 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. 2023-04-25 02:54:09,663 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 388bc1373cd34a10ba8be8ddc731e198: 2023-04-25 02:54:09,665 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-25 02:54:09,666 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682391249666"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682391249666"}]},"ts":"1682391249666"} 2023-04-25 02:54:09,667 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-25 02:54:09,668 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-25 02:54:09,668 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682391249668"}]},"ts":"1682391249668"} 2023-04-25 02:54:09,669 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-25 02:54:09,672 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=17, ppid=16, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=388bc1373cd34a10ba8be8ddc731e198, ASSIGN}] 2023-04-25 02:54:09,674 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=17, ppid=16, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=388bc1373cd34a10ba8be8ddc731e198, ASSIGN 2023-04-25 02:54:09,674 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=17, ppid=16, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=388bc1373cd34a10ba8be8ddc731e198, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,46697,1682391220023; forceNewPlan=false, retain=false 2023-04-25 02:54:09,825 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=17 updating hbase:meta row=388bc1373cd34a10ba8be8ddc731e198, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:09,826 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682391249825"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682391249825"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682391249825"}]},"ts":"1682391249825"} 2023-04-25 02:54:09,827 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=18, ppid=17, state=RUNNABLE; OpenRegionProcedure 388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023}] 2023-04-25 02:54:09,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-25 02:54:09,983 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. 2023-04-25 02:54:09,984 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 388bc1373cd34a10ba8be8ddc731e198, NAME => 'TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198.', STARTKEY => '', ENDKEY => ''} 2023-04-25 02:54:09,984 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 388bc1373cd34a10ba8be8ddc731e198 2023-04-25 02:54:09,984 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-25 02:54:09,984 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 388bc1373cd34a10ba8be8ddc731e198 2023-04-25 02:54:09,984 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 388bc1373cd34a10ba8be8ddc731e198 2023-04-25 02:54:09,985 INFO [StoreOpener-388bc1373cd34a10ba8be8ddc731e198-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 388bc1373cd34a10ba8be8ddc731e198 2023-04-25 02:54:09,987 DEBUG [StoreOpener-388bc1373cd34a10ba8be8ddc731e198-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/A 2023-04-25 02:54:09,987 DEBUG [StoreOpener-388bc1373cd34a10ba8be8ddc731e198-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/A 2023-04-25 02:54:09,987 INFO [StoreOpener-388bc1373cd34a10ba8be8ddc731e198-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-25 02:54:09,987 INFO [StoreOpener-388bc1373cd34a10ba8be8ddc731e198-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 388bc1373cd34a10ba8be8ddc731e198 columnFamilyName A 2023-04-25 02:54:09,988 INFO [StoreOpener-388bc1373cd34a10ba8be8ddc731e198-1] regionserver.HStore(310): Store=388bc1373cd34a10ba8be8ddc731e198/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-25 02:54:09,988 INFO [StoreOpener-388bc1373cd34a10ba8be8ddc731e198-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 388bc1373cd34a10ba8be8ddc731e198 2023-04-25 02:54:09,989 DEBUG [StoreOpener-388bc1373cd34a10ba8be8ddc731e198-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/B 2023-04-25 02:54:09,989 DEBUG [StoreOpener-388bc1373cd34a10ba8be8ddc731e198-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/B 2023-04-25 02:54:09,990 INFO [StoreOpener-388bc1373cd34a10ba8be8ddc731e198-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-25 02:54:09,990 INFO [StoreOpener-388bc1373cd34a10ba8be8ddc731e198-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 388bc1373cd34a10ba8be8ddc731e198 columnFamilyName B 2023-04-25 02:54:09,990 INFO [StoreOpener-388bc1373cd34a10ba8be8ddc731e198-1] regionserver.HStore(310): Store=388bc1373cd34a10ba8be8ddc731e198/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-25 02:54:09,990 INFO [StoreOpener-388bc1373cd34a10ba8be8ddc731e198-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 388bc1373cd34a10ba8be8ddc731e198 2023-04-25 02:54:09,991 DEBUG [StoreOpener-388bc1373cd34a10ba8be8ddc731e198-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/C 2023-04-25 02:54:09,992 DEBUG [StoreOpener-388bc1373cd34a10ba8be8ddc731e198-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/C 2023-04-25 02:54:09,992 INFO [StoreOpener-388bc1373cd34a10ba8be8ddc731e198-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-25 02:54:09,992 INFO [StoreOpener-388bc1373cd34a10ba8be8ddc731e198-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 388bc1373cd34a10ba8be8ddc731e198 columnFamilyName C 2023-04-25 02:54:09,992 INFO [StoreOpener-388bc1373cd34a10ba8be8ddc731e198-1] regionserver.HStore(310): Store=388bc1373cd34a10ba8be8ddc731e198/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-25 02:54:09,992 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. 2023-04-25 02:54:09,993 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198 2023-04-25 02:54:09,993 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198 2023-04-25 02:54:09,995 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-25 02:54:09,997 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 388bc1373cd34a10ba8be8ddc731e198 2023-04-25 02:54:09,998 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-25 02:54:09,999 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 388bc1373cd34a10ba8be8ddc731e198; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=73067111, jitterRate=0.08878479897975922}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-25 02:54:09,999 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 388bc1373cd34a10ba8be8ddc731e198: 2023-04-25 02:54:10,000 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198., pid=18, masterSystemTime=1682391249979 2023-04-25 02:54:10,001 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. 2023-04-25 02:54:10,001 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. 2023-04-25 02:54:10,002 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=17 updating hbase:meta row=388bc1373cd34a10ba8be8ddc731e198, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:10,002 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682391250002"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1682391250002"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1682391250002"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1682391250002"}]},"ts":"1682391250002"} 2023-04-25 02:54:10,005 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=18, resume processing ppid=17 2023-04-25 02:54:10,005 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=18, ppid=17, state=SUCCESS; OpenRegionProcedure 388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 in 176 msec 2023-04-25 02:54:10,006 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=17, resume processing ppid=16 2023-04-25 02:54:10,007 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=17, ppid=16, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=388bc1373cd34a10ba8be8ddc731e198, ASSIGN in 333 msec 2023-04-25 02:54:10,007 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-25 02:54:10,007 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682391250007"}]},"ts":"1682391250007"} 2023-04-25 02:54:10,008 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-25 02:54:10,010 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-25 02:54:10,011 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=16, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1720 sec 2023-04-25 02:54:10,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-25 02:54:10,947 INFO [Listener at localhost/41915] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 16 completed 2023-04-25 02:54:10,949 DEBUG [Listener at localhost/41915] zookeeper.ReadOnlyZKClient(139): Connect 0x324dc814 to 127.0.0.1:62054 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-25 02:54:10,955 DEBUG [Listener at localhost/41915] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5a301dcb, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-25 02:54:10,956 DEBUG [Listener at localhost/41915] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-25 02:54:10,960 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53606, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-25 02:54:10,961 DEBUG [Listener at localhost/41915] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-25 02:54:10,963 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46500, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-25 02:54:10,970 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-25 02:54:10,970 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.HMaster$12(2454): Client=jenkins//172.31.14.131 modify table TestAcidGuarantees from 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} to 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', MOB_THRESHOLD => '4', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-25 02:54:10,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure2.ProcedureExecutor(1029): Stored pid=19, state=RUNNABLE:MODIFY_TABLE_PREPARE; ModifyTableProcedure table=TestAcidGuarantees 2023-04-25 02:54:11,392 DEBUG [PEWorker-4] util.FSTableDescriptors(570): Wrote into hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-25 02:54:11,393 DEBUG [PEWorker-4] util.FSTableDescriptors(519): Deleted hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-25 02:54:11,393 INFO [PEWorker-4] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-25 02:54:11,397 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=20, ppid=19, state=RUNNABLE:REOPEN_TABLE_REGIONS_GET_REGIONS; ReopenTableRegionsProcedure table=TestAcidGuarantees}] 2023-04-25 02:54:11,416 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=21, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=388bc1373cd34a10ba8be8ddc731e198, REOPEN/MOVE}] 2023-04-25 02:54:11,417 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=21, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=388bc1373cd34a10ba8be8ddc731e198, REOPEN/MOVE 2023-04-25 02:54:11,418 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=388bc1373cd34a10ba8be8ddc731e198, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:11,418 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682391251417"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682391251417"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682391251417"}]},"ts":"1682391251417"} 2023-04-25 02:54:11,419 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=22, ppid=21, state=RUNNABLE; CloseRegionProcedure 388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023}] 2023-04-25 02:54:11,572 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 388bc1373cd34a10ba8be8ddc731e198 2023-04-25 02:54:11,573 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 388bc1373cd34a10ba8be8ddc731e198, disabling compactions & flushes 2023-04-25 02:54:11,573 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. 2023-04-25 02:54:11,573 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. 2023-04-25 02:54:11,573 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. after waiting 0 ms 2023-04-25 02:54:11,573 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. 2023-04-25 02:54:11,577 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-04-25 02:54:11,577 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. 2023-04-25 02:54:11,577 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 388bc1373cd34a10ba8be8ddc731e198: 2023-04-25 02:54:11,577 WARN [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(3509): Not adding moved region record: 388bc1373cd34a10ba8be8ddc731e198 to self. 2023-04-25 02:54:11,579 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 388bc1373cd34a10ba8be8ddc731e198 2023-04-25 02:54:11,579 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=388bc1373cd34a10ba8be8ddc731e198, regionState=CLOSED 2023-04-25 02:54:11,579 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682391251579"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682391251579"}]},"ts":"1682391251579"} 2023-04-25 02:54:11,582 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=22, resume processing ppid=21 2023-04-25 02:54:11,582 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=22, ppid=21, state=SUCCESS; CloseRegionProcedure 388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 in 162 msec 2023-04-25 02:54:11,583 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=21, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=388bc1373cd34a10ba8be8ddc731e198, REOPEN/MOVE; state=CLOSED, location=jenkins-hbase4.apache.org,46697,1682391220023; forceNewPlan=false, retain=true 2023-04-25 02:54:11,733 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=388bc1373cd34a10ba8be8ddc731e198, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:11,733 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682391251733"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682391251733"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682391251733"}]},"ts":"1682391251733"} 2023-04-25 02:54:11,735 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=23, ppid=21, state=RUNNABLE; OpenRegionProcedure 388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023}] 2023-04-25 02:54:11,891 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. 2023-04-25 02:54:11,891 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 388bc1373cd34a10ba8be8ddc731e198, NAME => 'TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198.', STARTKEY => '', ENDKEY => ''} 2023-04-25 02:54:11,892 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 388bc1373cd34a10ba8be8ddc731e198 2023-04-25 02:54:11,892 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-25 02:54:11,892 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 388bc1373cd34a10ba8be8ddc731e198 2023-04-25 02:54:11,892 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 388bc1373cd34a10ba8be8ddc731e198 2023-04-25 02:54:11,894 INFO [StoreOpener-388bc1373cd34a10ba8be8ddc731e198-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 388bc1373cd34a10ba8be8ddc731e198 2023-04-25 02:54:11,895 DEBUG [StoreOpener-388bc1373cd34a10ba8be8ddc731e198-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/A 2023-04-25 02:54:11,895 DEBUG [StoreOpener-388bc1373cd34a10ba8be8ddc731e198-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/A 2023-04-25 02:54:11,896 INFO [StoreOpener-388bc1373cd34a10ba8be8ddc731e198-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-25 02:54:11,898 INFO [StoreOpener-388bc1373cd34a10ba8be8ddc731e198-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 388bc1373cd34a10ba8be8ddc731e198 columnFamilyName A 2023-04-25 02:54:11,900 INFO [StoreOpener-388bc1373cd34a10ba8be8ddc731e198-1] regionserver.HStore(310): Store=388bc1373cd34a10ba8be8ddc731e198/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-25 02:54:11,901 INFO [StoreOpener-388bc1373cd34a10ba8be8ddc731e198-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 388bc1373cd34a10ba8be8ddc731e198 2023-04-25 02:54:11,902 DEBUG [StoreOpener-388bc1373cd34a10ba8be8ddc731e198-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/B 2023-04-25 02:54:11,902 DEBUG [StoreOpener-388bc1373cd34a10ba8be8ddc731e198-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/B 2023-04-25 02:54:11,902 INFO [StoreOpener-388bc1373cd34a10ba8be8ddc731e198-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-25 02:54:11,902 INFO [StoreOpener-388bc1373cd34a10ba8be8ddc731e198-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 388bc1373cd34a10ba8be8ddc731e198 columnFamilyName B 2023-04-25 02:54:11,903 INFO [StoreOpener-388bc1373cd34a10ba8be8ddc731e198-1] regionserver.HStore(310): Store=388bc1373cd34a10ba8be8ddc731e198/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-25 02:54:11,903 INFO [StoreOpener-388bc1373cd34a10ba8be8ddc731e198-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 388bc1373cd34a10ba8be8ddc731e198 2023-04-25 02:54:11,904 DEBUG [StoreOpener-388bc1373cd34a10ba8be8ddc731e198-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/C 2023-04-25 02:54:11,904 DEBUG [StoreOpener-388bc1373cd34a10ba8be8ddc731e198-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/C 2023-04-25 02:54:11,904 INFO [StoreOpener-388bc1373cd34a10ba8be8ddc731e198-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-25 02:54:11,904 INFO [StoreOpener-388bc1373cd34a10ba8be8ddc731e198-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 388bc1373cd34a10ba8be8ddc731e198 columnFamilyName C 2023-04-25 02:54:11,905 INFO [StoreOpener-388bc1373cd34a10ba8be8ddc731e198-1] regionserver.HStore(310): Store=388bc1373cd34a10ba8be8ddc731e198/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-25 02:54:11,905 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. 2023-04-25 02:54:11,906 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198 2023-04-25 02:54:11,907 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198 2023-04-25 02:54:11,909 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-25 02:54:11,910 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 388bc1373cd34a10ba8be8ddc731e198 2023-04-25 02:54:11,911 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 388bc1373cd34a10ba8be8ddc731e198; next sequenceid=5; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=73449484, jitterRate=0.09448260068893433}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-25 02:54:11,911 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 388bc1373cd34a10ba8be8ddc731e198: 2023-04-25 02:54:11,912 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198., pid=23, masterSystemTime=1682391251887 2023-04-25 02:54:11,913 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. 2023-04-25 02:54:11,913 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. 2023-04-25 02:54:11,914 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=388bc1373cd34a10ba8be8ddc731e198, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:11,914 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682391251914"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1682391251914"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1682391251914"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1682391251914"}]},"ts":"1682391251914"} 2023-04-25 02:54:11,917 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=23, resume processing ppid=21 2023-04-25 02:54:11,917 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=23, ppid=21, state=SUCCESS; OpenRegionProcedure 388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 in 180 msec 2023-04-25 02:54:11,918 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=21, resume processing ppid=20 2023-04-25 02:54:11,919 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=21, ppid=20, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=388bc1373cd34a10ba8be8ddc731e198, REOPEN/MOVE in 501 msec 2023-04-25 02:54:11,921 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=20, resume processing ppid=19 2023-04-25 02:54:11,921 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=20, ppid=19, state=SUCCESS; ReopenTableRegionsProcedure table=TestAcidGuarantees in 524 msec 2023-04-25 02:54:11,923 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=19, state=SUCCESS; ModifyTableProcedure table=TestAcidGuarantees in 949 msec 2023-04-25 02:54:11,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1227): Checking to see if procedure is done pid=19 2023-04-25 02:54:11,932 DEBUG [Listener at localhost/41915] zookeeper.ReadOnlyZKClient(139): Connect 0x55cbb115 to 127.0.0.1:62054 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-25 02:54:11,937 DEBUG [Listener at localhost/41915] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1aa5eef5, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-25 02:54:11,938 DEBUG [Listener at localhost/41915] zookeeper.ReadOnlyZKClient(139): Connect 0x24bd3872 to 127.0.0.1:62054 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-25 02:54:11,942 DEBUG [Listener at localhost/41915] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@73e51571, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-25 02:54:11,942 DEBUG [Listener at localhost/41915] zookeeper.ReadOnlyZKClient(139): Connect 0x7aba8e46 to 127.0.0.1:62054 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-25 02:54:11,947 DEBUG [Listener at localhost/41915] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@370947cb, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-25 02:54:11,948 DEBUG [Listener at localhost/41915] zookeeper.ReadOnlyZKClient(139): Connect 0x210d1008 to 127.0.0.1:62054 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-25 02:54:11,955 DEBUG [Listener at localhost/41915] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5c472cfd, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-25 02:54:11,956 DEBUG [Listener at localhost/41915] zookeeper.ReadOnlyZKClient(139): Connect 0x00c2598b to 127.0.0.1:62054 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-25 02:54:11,961 DEBUG [Listener at localhost/41915] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@498694cb, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-25 02:54:11,961 DEBUG [Listener at localhost/41915] zookeeper.ReadOnlyZKClient(139): Connect 0x421e5356 to 127.0.0.1:62054 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-25 02:54:11,978 DEBUG [Listener at localhost/41915] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3f43b712, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-25 02:54:11,979 DEBUG [Listener at localhost/41915] zookeeper.ReadOnlyZKClient(139): Connect 0x4a0fd7b4 to 127.0.0.1:62054 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-25 02:54:11,991 DEBUG [Listener at localhost/41915] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7a3bb940, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-25 02:54:11,991 DEBUG [Listener at localhost/41915] zookeeper.ReadOnlyZKClient(139): Connect 0x1a7b66a3 to 127.0.0.1:62054 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-25 02:54:11,997 DEBUG [Listener at localhost/41915] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2efdb02f, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-25 02:54:11,998 DEBUG [Listener at localhost/41915] zookeeper.ReadOnlyZKClient(139): Connect 0x42d325f4 to 127.0.0.1:62054 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-25 02:54:12,013 DEBUG [Listener at localhost/41915] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3dc1d73, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-25 02:54:12,032 DEBUG [hconnection-0x49a54980-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-25 02:54:12,036 DEBUG [hconnection-0x44a2174c-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-25 02:54:12,038 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53610, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-25 02:54:12,040 DEBUG [hconnection-0x4f1c6982-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-25 02:54:12,052 DEBUG [hconnection-0x4fedaced-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-25 02:54:12,053 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53642, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-25 02:54:12,065 DEBUG [hconnection-0x25fd580c-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-25 02:54:12,065 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53618, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-25 02:54:12,065 DEBUG [hconnection-0x5dcd6e6-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-25 02:54:12,065 DEBUG [hconnection-0x5246c436-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-25 02:54:12,065 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53632, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-25 02:54:12,068 DEBUG [hconnection-0x53b42b0-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-25 02:54:12,077 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53650, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-25 02:54:12,071 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53664, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-25 02:54:12,078 DEBUG [hconnection-0x28e306bd-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-25 02:54:12,080 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53690, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-25 02:54:12,083 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:54:12,089 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53678, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-25 02:54:12,102 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53692, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-25 02:54:12,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:54:12,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:54:12,108 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:54:12,108 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:12,109 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:54:12,109 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:54:12,110 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:12,110 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:12,111 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:12,111 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:12,111 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:12,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:12,112 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:12,112 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:54:12,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:12,112 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:12,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:54:12,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:12,113 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:54:12,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(9158): Flush requested on 388bc1373cd34a10ba8be8ddc731e198 2023-04-25 02:54:12,117 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 388bc1373cd34a10ba8be8ddc731e198 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-25 02:54:12,117 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:54:12,117 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:54:12,117 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:54:12,117 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:12,117 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:54:12,118 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 388bc1373cd34a10ba8be8ddc731e198, store=A 2023-04-25 02:54:12,118 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:54:12,118 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 388bc1373cd34a10ba8be8ddc731e198, store=B 2023-04-25 02:54:12,118 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:54:12,118 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 388bc1373cd34a10ba8be8ddc731e198, store=C 2023-04-25 02:54:12,118 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:54:12,126 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:54:12,126 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:54:12,126 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. 2023-04-25 02:54:12,126 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. started... 2023-04-25 02:54:12,126 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. as already flushing 2023-04-25 02:54:12,126 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. 2023-04-25 02:54:12,126 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:54:12,126 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:54:12,126 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:12,135 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:54:12,135 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:54:12,136 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:12,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:12,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:12,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:12,137 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:12,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:12,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:12,137 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:12,137 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:54:12,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:12,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:12,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:12,140 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:54:12,140 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@36e64453[Count = 0] remaining members to acquire global barrier 2023-04-25 02:54:12,140 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:54:12,140 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:12,141 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1682391252128, max timestamp=1682391252138Num uniques 859; 2023-04-25 02:54:12,141 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-25 02:54:12,142 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:12,142 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:12,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:12,142 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:54:12,142 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:54:12,142 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:54:12,142 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682391252128, max timestamp=1682391252140Num uniques 859; 2023-04-25 02:54:12,142 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:12,143 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:54:12,143 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-25 02:54:12,143 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682391252128, max timestamp=1682391252140Num uniques 859; 2023-04-25 02:54:12,143 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-25 02:54:12,144 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:54:12,144 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:12,144 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:54:12,144 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:12,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:12,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:12,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:12,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:12,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:12,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:12,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:12,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:12,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:12,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:12,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:54:12,148 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:54:12,148 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:54:12,148 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:54:12,148 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:54:12,148 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:54:12,148 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:54:12,150 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:12,150 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:12,150 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:12,150 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:12,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:12,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:12,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:12,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:12,151 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:12,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:12,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:12,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:12,152 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:12,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:12,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:12,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:12,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:12,155 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:12,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:12,155 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230425e381582adbf148c3af83695597a589f2 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425e381582adbf148c3af83695597a589f2 2023-04-25 02:54:12,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:12,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:12,161 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:12,161 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:12,161 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:12,162 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:12,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:12,162 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:54:12,162 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:54:12,161 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:12,162 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:12,162 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:12,162 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:12,162 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:12,162 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:12,162 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:12,162 DEBUG [Thread-799] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:54:12,162 DEBUG [Thread-799] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:54:12,163 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:12,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:12,170 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:12,170 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:12,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53610 deadline: 1682391312162, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:12,170 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:12,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53642 deadline: 1682391312168, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:12,171 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:12,171 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53650 deadline: 1682391312170, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:12,171 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:12,171 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53692 deadline: 1682391312170, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:12,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53664 deadline: 1682391312164, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:12,173 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/.tmp/A/526ce0eef8a04f3a8c1b08316b8dc391 2023-04-25 02:54:12,217 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=15 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/.tmp/B/9d672d5b816b4f128fd34506fb1fa281 2023-04-25 02:54:12,258 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=15 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/.tmp/C/ea45787746b740a88796fac1b681bb91 2023-04-25 02:54:12,262 DEBUG [Thread-799] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:54:12,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:54:12,268 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:54:12,268 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/.tmp/A/526ce0eef8a04f3a8c1b08316b8dc391 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/A/526ce0eef8a04f3a8c1b08316b8dc391 2023-04-25 02:54:12,274 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:12,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53610 deadline: 1682391312272, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:12,274 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:12,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53650 deadline: 1682391312272, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:12,275 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:12,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53642 deadline: 1682391312273, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:12,275 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:12,275 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:12,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53664 deadline: 1682391312274, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:12,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53692 deadline: 1682391312274, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:12,279 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/A/526ce0eef8a04f3a8c1b08316b8dc391, entries=150, sequenceid=15, filesize=25.1 K 2023-04-25 02:54:12,281 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/.tmp/B/9d672d5b816b4f128fd34506fb1fa281 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/B/9d672d5b816b4f128fd34506fb1fa281 2023-04-25 02:54:12,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:54:12,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:54:12,283 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:54:12,283 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:12,283 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:54:12,283 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:54:12,284 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:12,284 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:12,285 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:12,285 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:12,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:12,285 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:12,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:12,286 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:12,286 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:54:12,286 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:12,286 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/B/9d672d5b816b4f128fd34506fb1fa281, entries=150, sequenceid=15, filesize=11.6 K 2023-04-25 02:54:12,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:54:12,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:12,286 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:54:12,287 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/.tmp/C/ea45787746b740a88796fac1b681bb91 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/C/ea45787746b740a88796fac1b681bb91 2023-04-25 02:54:12,291 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/C/ea45787746b740a88796fac1b681bb91, entries=150, sequenceid=15, filesize=11.6 K 2023-04-25 02:54:12,292 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:54:12,292 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:54:12,292 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:54:12,292 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:12,292 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:54:12,292 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 388bc1373cd34a10ba8be8ddc731e198 in 175ms, sequenceid=15, compaction requested=false 2023-04-25 02:54:12,292 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 388bc1373cd34a10ba8be8ddc731e198: 2023-04-25 02:54:12,300 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:54:12,300 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:54:12,300 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. 2023-04-25 02:54:12,300 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. started... 2023-04-25 02:54:12,300 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 388bc1373cd34a10ba8be8ddc731e198 3/3 column families, dataSize=127.47 KB heapSize=360.38 KB 2023-04-25 02:54:12,301 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 388bc1373cd34a10ba8be8ddc731e198, store=A 2023-04-25 02:54:12,301 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 02:54:12,301 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 388bc1373cd34a10ba8be8ddc731e198, store=B 2023-04-25 02:54:12,301 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 02:54:12,301 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 388bc1373cd34a10ba8be8ddc731e198, store=C 2023-04-25 02:54:12,301 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 02:54:12,382 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230425a6a8b50d295b4aebb142f3803176ed25 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425a6a8b50d295b4aebb142f3803176ed25 2023-04-25 02:54:12,407 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=50, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/.tmp/A/25246c3480d543cea2f7d1c2b9eb48f8 2023-04-25 02:54:12,480 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(9158): Flush requested on 388bc1373cd34a10ba8be8ddc731e198 2023-04-25 02:54:12,480 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. as already flushing 2023-04-25 02:54:12,489 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:12,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53664 deadline: 1682391312484, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:12,489 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:12,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53642 deadline: 1682391312486, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:12,491 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:12,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53650 deadline: 1682391312487, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:12,493 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:12,493 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53610 deadline: 1682391312489, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:12,493 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:12,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53692 deadline: 1682391312489, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:12,591 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:12,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53642 deadline: 1682391312590, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:12,591 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:12,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53664 deadline: 1682391312590, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:12,596 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:12,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53650 deadline: 1682391312593, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:12,596 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:12,596 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:12,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53610 deadline: 1682391312596, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:12,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53692 deadline: 1682391312595, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:12,795 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:12,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53642 deadline: 1682391312794, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:12,795 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:12,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53664 deadline: 1682391312794, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:12,798 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:12,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53650 deadline: 1682391312797, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:12,799 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:12,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53692 deadline: 1682391312798, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:12,799 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:12,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53610 deadline: 1682391312798, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:12,847 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=50 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/.tmp/B/e411495647fd4bfe994149ae41b61d23 2023-04-25 02:54:12,873 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=50 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/.tmp/C/d0dcc153e64a47cd9669afca16d2538f 2023-04-25 02:54:12,880 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/.tmp/A/25246c3480d543cea2f7d1c2b9eb48f8 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/A/25246c3480d543cea2f7d1c2b9eb48f8 2023-04-25 02:54:12,885 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/A/25246c3480d543cea2f7d1c2b9eb48f8, entries=150, sequenceid=50, filesize=25.1 K 2023-04-25 02:54:12,886 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/.tmp/B/e411495647fd4bfe994149ae41b61d23 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/B/e411495647fd4bfe994149ae41b61d23 2023-04-25 02:54:12,895 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/B/e411495647fd4bfe994149ae41b61d23, entries=150, sequenceid=50, filesize=11.6 K 2023-04-25 02:54:12,896 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/.tmp/C/d0dcc153e64a47cd9669afca16d2538f as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/C/d0dcc153e64a47cd9669afca16d2538f 2023-04-25 02:54:12,903 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/C/d0dcc153e64a47cd9669afca16d2538f, entries=150, sequenceid=50, filesize=11.6 K 2023-04-25 02:54:12,904 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~360.33 KB/368976, currentSize=60.38 KB/61830 for 388bc1373cd34a10ba8be8ddc731e198 in 604ms, sequenceid=50, compaction requested=false 2023-04-25 02:54:12,904 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 388bc1373cd34a10ba8be8ddc731e198: 2023-04-25 02:54:12,904 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. 2023-04-25 02:54:12,904 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:54:12,904 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:54:12,904 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:12,904 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:54:12,904 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:54:12,906 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:12,906 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:12,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:12,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:12,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:12,906 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:12,906 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:54:12,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:12,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:12,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:12,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:12,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:12,908 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:54:12,908 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5fb49c52[Count = 0] remaining members to acquire global barrier 2023-04-25 02:54:12,908 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:54:12,908 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:12,910 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:12,910 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:12,910 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:12,910 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:54:12,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:12,910 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:54:12,910 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:54:12,910 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:54:12,915 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:54:12,915 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:12,915 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:12,915 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:54:12,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:12,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:12,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:12,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:12,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:12,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:12,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:12,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:12,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:12,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:12,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:54:12,917 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:54:12,917 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:54:12,917 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:54:12,917 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:54:12,917 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:54:12,917 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:54:12,919 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:12,919 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:12,919 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:12,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:12,920 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:12,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:12,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:12,920 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:12,923 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:12,923 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:12,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:12,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:12,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:12,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:12,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:12,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:12,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:12,927 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:12,927 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:12,927 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:12,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:12,928 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:12,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:12,928 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:12,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:12,928 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:54:12,928 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:54:12,928 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:12,928 DEBUG [Thread-799] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:54:12,928 DEBUG [Thread-799] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:54:12,928 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:12,929 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:12,929 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:12,929 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:12,929 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:12,929 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:12,929 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:54:13,029 DEBUG [Thread-799] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:54:13,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:54:13,036 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:54:13,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:54:13,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:54:13,042 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:54:13,042 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:13,042 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:54:13,042 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:54:13,043 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:13,043 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:13,044 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:13,044 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:13,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:13,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:13,045 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:13,045 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:13,045 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:13,045 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:54:13,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:54:13,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:13,046 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:54:13,052 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:54:13,052 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:54:13,052 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:54:13,052 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:13,052 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:54:13,053 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:54:13,053 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:54:13,053 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. 2023-04-25 02:54:13,053 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. started... 2023-04-25 02:54:13,053 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 388bc1373cd34a10ba8be8ddc731e198 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-25 02:54:13,053 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 388bc1373cd34a10ba8be8ddc731e198, store=A 2023-04-25 02:54:13,054 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:54:13,054 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 388bc1373cd34a10ba8be8ddc731e198, store=B 2023-04-25 02:54:13,054 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:54:13,054 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 388bc1373cd34a10ba8be8ddc731e198, store=C 2023-04-25 02:54:13,054 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:54:13,083 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230425f3fff75d996540109428076c602f32ed to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425f3fff75d996540109428076c602f32ed 2023-04-25 02:54:13,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(9158): Flush requested on 388bc1373cd34a10ba8be8ddc731e198 2023-04-25 02:54:13,099 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. as already flushing 2023-04-25 02:54:13,125 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682391253112, max timestamp=1682391253123Num uniques 859; 2023-04-25 02:54:13,125 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-25 02:54:13,125 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682391253112, max timestamp=1682391253123Num uniques 859; 2023-04-25 02:54:13,125 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-25 02:54:13,126 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1682391253114, max timestamp=1682391253123Num uniques 859; 2023-04-25 02:54:13,127 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-25 02:54:13,148 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:13,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53692 deadline: 1682391313141, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:13,149 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:13,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53642 deadline: 1682391313142, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:13,151 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:13,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53610 deadline: 1682391313143, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:13,151 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:13,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53650 deadline: 1682391313143, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:13,152 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:13,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53664 deadline: 1682391313145, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:13,255 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:13,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53692 deadline: 1682391313250, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:13,256 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:13,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53642 deadline: 1682391313251, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:13,257 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:13,257 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:13,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53610 deadline: 1682391313253, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:13,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53650 deadline: 1682391313253, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:13,257 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:13,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53664 deadline: 1682391313254, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:13,462 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:13,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53692 deadline: 1682391313457, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:13,463 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:13,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53642 deadline: 1682391313458, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:13,463 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:13,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53664 deadline: 1682391313459, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:13,464 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:13,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53610 deadline: 1682391313459, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:13,464 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:13,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53650 deadline: 1682391313459, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:13,503 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=62, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/.tmp/A/7c2d5bcd4cbd44eb9aa87771ea9ec9ac 2023-04-25 02:54:13,535 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=62 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/.tmp/B/2500b86ed0524a3488809f7eb6811311 2023-04-25 02:54:13,561 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=62 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/.tmp/C/d4d62a806d6a48a380c0e88b0525fca1 2023-04-25 02:54:13,571 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/.tmp/A/7c2d5bcd4cbd44eb9aa87771ea9ec9ac as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/A/7c2d5bcd4cbd44eb9aa87771ea9ec9ac 2023-04-25 02:54:13,578 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/A/7c2d5bcd4cbd44eb9aa87771ea9ec9ac, entries=150, sequenceid=62, filesize=25.1 K 2023-04-25 02:54:13,579 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/.tmp/B/2500b86ed0524a3488809f7eb6811311 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/B/2500b86ed0524a3488809f7eb6811311 2023-04-25 02:54:13,589 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/B/2500b86ed0524a3488809f7eb6811311, entries=150, sequenceid=62, filesize=11.6 K 2023-04-25 02:54:13,590 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/.tmp/C/d4d62a806d6a48a380c0e88b0525fca1 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/C/d4d62a806d6a48a380c0e88b0525fca1 2023-04-25 02:54:13,596 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/C/d4d62a806d6a48a380c0e88b0525fca1, entries=150, sequenceid=62, filesize=11.6 K 2023-04-25 02:54:13,597 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=125.23 KB/128240 for 388bc1373cd34a10ba8be8ddc731e198 in 544ms, sequenceid=62, compaction requested=true 2023-04-25 02:54:13,597 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 388bc1373cd34a10ba8be8ddc731e198: 2023-04-25 02:54:13,597 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. 2023-04-25 02:54:13,597 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:54:13,597 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:54:13,597 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:13,601 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:54:13,601 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:54:13,604 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:13,604 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:13,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:13,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:13,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:13,604 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:13,605 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:54:13,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:13,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:13,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:13,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:13,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:13,606 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:54:13,606 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6e8d67ea[Count = 0] remaining members to acquire global barrier 2023-04-25 02:54:13,606 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:54:13,606 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:13,608 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:13,608 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:13,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:13,608 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:54:13,608 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:54:13,608 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:54:13,608 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:13,608 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:54:13,610 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:13,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:13,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:13,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:13,610 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:54:13,610 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:13,611 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:54:13,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:13,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:13,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:13,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:13,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:13,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:13,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:13,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:54:13,613 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:54:13,613 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:54:13,613 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:54:13,613 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:54:13,613 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:54:13,613 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:54:13,615 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:13,615 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:13,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:13,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:13,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:13,615 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:13,615 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:13,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:13,615 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:13,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:13,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:13,616 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:13,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:13,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:13,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:13,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:13,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:13,619 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:13,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:13,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:13,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:13,626 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:13,626 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:13,626 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:13,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:13,626 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:13,626 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:13,626 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:54:13,626 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:54:13,626 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:13,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:13,626 DEBUG [Thread-799] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:54:13,626 DEBUG [Thread-799] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:54:13,626 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:13,626 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:13,627 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:13,627 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:13,627 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:13,627 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:13,727 DEBUG [Thread-799] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:54:13,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:54:13,736 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:54:13,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:54:13,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:54:13,753 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:54:13,753 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:13,753 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:54:13,753 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:54:13,753 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:13,753 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:13,755 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:13,755 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:13,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:13,755 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:13,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:13,756 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:13,756 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:54:13,756 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:13,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:54:13,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:13,757 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:54:13,760 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:54:13,760 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:54:13,760 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:54:13,760 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:13,760 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:54:13,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(9158): Flush requested on 388bc1373cd34a10ba8be8ddc731e198 2023-04-25 02:54:13,768 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 388bc1373cd34a10ba8be8ddc731e198 3/3 column families, dataSize=131.94 KB heapSize=370.53 KB 2023-04-25 02:54:13,769 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 388bc1373cd34a10ba8be8ddc731e198, store=A 2023-04-25 02:54:13,769 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 02:54:13,769 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 388bc1373cd34a10ba8be8ddc731e198, store=B 2023-04-25 02:54:13,769 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 02:54:13,769 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 388bc1373cd34a10ba8be8ddc731e198, store=C 2023-04-25 02:54:13,770 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 02:54:13,772 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:54:13,772 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:54:13,772 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. 2023-04-25 02:54:13,772 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. started... 2023-04-25 02:54:13,772 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. as already flushing 2023-04-25 02:54:13,772 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. 2023-04-25 02:54:13,772 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:54:13,773 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:54:13,773 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:13,773 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:54:13,773 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:54:13,774 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:13,774 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:13,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:13,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:13,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:13,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:13,777 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:13,777 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:54:13,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:13,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:13,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:13,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:13,780 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:54:13,780 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@19d2888e[Count = 0] remaining members to acquire global barrier 2023-04-25 02:54:13,780 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:54:13,780 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:13,783 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:13,783 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:13,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:13,783 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:54:13,783 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:13,783 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:54:13,783 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:54:13,783 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:54:13,785 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:13,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53610 deadline: 1682391313776, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:13,785 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:13,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53692 deadline: 1682391313779, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:13,786 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:13,787 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:54:13,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:13,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:13,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:13,787 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:13,787 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:54:13,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:13,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:13,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:13,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:13,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:13,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:13,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:13,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:54:13,789 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:54:13,789 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:54:13,790 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:54:13,790 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:54:13,790 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:54:13,790 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:54:13,791 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:13,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53664 deadline: 1682391313784, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:13,792 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:13,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53650 deadline: 1682391313785, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:13,792 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:13,792 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:13,792 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:13,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:13,793 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:13,793 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:13,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:13,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:13,793 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:13,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:13,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:13,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:13,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:13,794 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:13,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:13,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:13,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:13,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:13,796 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:13,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:13,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:13,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:13,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53642 deadline: 1682391313786, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:13,799 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:13,799 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:13,799 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:13,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:13,800 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:13,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:13,800 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:13,800 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:13,800 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:13,800 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:13,800 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:13,800 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:54:13,800 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:54:13,800 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:13,801 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:13,801 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:13,801 DEBUG [Thread-799] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:54:13,801 DEBUG [Thread-799] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:54:13,824 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023042595e83d6c0fb349a1976f5ca5e5c75814 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042595e83d6c0fb349a1976f5ca5e5c75814 2023-04-25 02:54:13,892 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:13,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53692 deadline: 1682391313887, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:13,892 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:13,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53610 deadline: 1682391313887, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:13,900 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:13,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53650 deadline: 1682391313894, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:13,900 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:13,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53664 deadline: 1682391313895, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:13,900 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:13,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53642 deadline: 1682391313899, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:13,901 DEBUG [Thread-799] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:54:13,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:54:13,904 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:54:13,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:54:13,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:54:13,926 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:54:13,926 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:13,926 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:54:13,926 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:54:13,927 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:13,927 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:13,931 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:13,932 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:13,932 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:13,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:13,932 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:13,932 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:54:13,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:13,933 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:13,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:54:13,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:13,933 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:54:13,934 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:54:13,934 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:54:13,934 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:54:13,934 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:13,934 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:54:13,936 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:54:13,936 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:54:13,936 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. 2023-04-25 02:54:13,936 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. started... 2023-04-25 02:54:13,936 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. as already flushing 2023-04-25 02:54:13,936 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. 2023-04-25 02:54:13,936 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:54:13,936 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:54:13,936 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:13,936 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:54:13,936 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:54:13,938 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:13,938 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:13,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:13,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:13,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:13,939 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:13,939 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:54:13,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:13,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:13,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:13,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:13,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:13,941 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:54:13,941 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1d12933b[Count = 0] remaining members to acquire global barrier 2023-04-25 02:54:13,941 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:54:13,942 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:13,943 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:13,943 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:13,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:13,943 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:54:13,943 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:54:13,943 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:54:13,943 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:13,943 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:54:13,945 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:54:13,945 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:13,945 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:13,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:13,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:13,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:13,946 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:54:13,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:13,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:13,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:13,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:13,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:13,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:13,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:13,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:54:13,949 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:54:13,949 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:54:13,949 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:54:13,949 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:54:13,949 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:54:13,949 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:54:13,951 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:13,951 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:13,952 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:13,952 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:13,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:13,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:13,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:13,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:13,952 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:13,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:13,952 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:13,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:13,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:13,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:13,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:13,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:13,953 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:13,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:13,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:13,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:13,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:13,956 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:13,956 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:13,956 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:13,956 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:13,956 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:13,956 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:13,956 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:13,956 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:13,957 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:13,956 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:13,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:13,956 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:54:13,956 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:13,956 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:54:13,957 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:54:13,957 DEBUG [Thread-799] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:54:13,957 DEBUG [Thread-799] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:54:13,958 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:13,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:14,023 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-25 02:54:14,057 DEBUG [Thread-799] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:54:14,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:54:14,060 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:54:14,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:54:14,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:54:14,080 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:54:14,080 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:14,080 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:54:14,080 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:54:14,081 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:14,081 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:14,082 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:14,082 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:14,082 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:14,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:14,082 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:14,082 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:54:14,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:14,083 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:14,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:54:14,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:14,083 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:54:14,084 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:54:14,084 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:54:14,084 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:54:14,084 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:14,084 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:54:14,088 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:54:14,088 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:54:14,088 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. 2023-04-25 02:54:14,088 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. started... 2023-04-25 02:54:14,088 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. as already flushing 2023-04-25 02:54:14,088 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. 2023-04-25 02:54:14,088 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:54:14,088 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:54:14,088 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:14,088 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:54:14,088 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:54:14,090 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:14,090 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:14,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:14,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:14,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:14,091 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:14,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:14,091 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:54:14,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:14,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:14,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:14,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:14,092 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:54:14,092 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@663fcac7[Count = 0] remaining members to acquire global barrier 2023-04-25 02:54:14,092 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:54:14,092 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:14,093 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:14,093 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:14,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:14,094 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:14,094 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:54:14,094 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:54:14,094 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:54:14,094 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:54:14,095 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:54:14,095 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:14,095 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:14,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:14,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:14,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:14,095 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:54:14,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:14,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:14,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:14,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:14,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:14,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:14,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:14,097 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:14,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53692 deadline: 1682391314093, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:14,097 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:14,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53610 deadline: 1682391314094, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:14,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:54:14,099 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:54:14,099 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:54:14,099 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:54:14,099 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:54:14,099 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:54:14,099 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:54:14,100 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:14,101 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:14,101 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:14,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:14,101 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:14,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:14,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:14,104 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:14,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53664 deadline: 1682391314102, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:14,108 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:14,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:14,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:14,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:14,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:14,109 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:14,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:14,109 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:14,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53642 deadline: 1682391314103, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:14,109 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:14,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53650 deadline: 1682391314104, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:14,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:14,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:14,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:14,112 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:14,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:14,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:14,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:14,117 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:14,117 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:14,119 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:14,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:14,119 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:14,119 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:14,119 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:14,119 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:14,119 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:14,119 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:14,119 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:54:14,119 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:14,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:14,119 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:14,119 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:14,119 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:54:14,120 DEBUG [Thread-799] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:54:14,120 DEBUG [Thread-799] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:54:14,220 DEBUG [Thread-799] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:54:14,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:54:14,223 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:54:14,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:54:14,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:54:14,224 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:54:14,224 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:14,224 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:54:14,224 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:54:14,225 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:14,225 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:14,226 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:14,226 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:14,226 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:14,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:14,227 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:14,227 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:54:14,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:14,227 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:14,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:54:14,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:14,228 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:54:14,232 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:54:14,232 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:54:14,232 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:54:14,232 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:14,232 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:54:14,232 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:54:14,232 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:54:14,232 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. 2023-04-25 02:54:14,233 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. started... 2023-04-25 02:54:14,233 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. as already flushing 2023-04-25 02:54:14,233 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. 2023-04-25 02:54:14,233 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:54:14,233 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:54:14,233 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:14,233 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:54:14,233 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:54:14,234 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:14,234 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:14,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:14,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:14,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:14,235 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:14,235 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:54:14,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:14,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:14,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:14,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:14,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:14,236 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:54:14,236 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@630843dd[Count = 0] remaining members to acquire global barrier 2023-04-25 02:54:14,236 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:54:14,236 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:14,237 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:14,237 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:14,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:14,237 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:54:14,237 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:54:14,238 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:54:14,238 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:14,238 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:54:14,240 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:14,240 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:54:14,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:14,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:14,240 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:14,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:14,240 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:54:14,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:14,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:14,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:14,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:14,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:14,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:14,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:14,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:54:14,243 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:54:14,243 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:54:14,243 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:54:14,243 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:54:14,243 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:54:14,243 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:54:14,246 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:14,246 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:14,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:14,247 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:14,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:14,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:14,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:14,246 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:14,247 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:14,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:14,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:14,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:14,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:14,247 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:14,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:14,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:14,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:14,250 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:14,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:14,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:14,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:14,258 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:14,258 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:14,258 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:14,258 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:14,258 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:54:14,258 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:54:14,258 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:14,258 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:14,258 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:14,258 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:14,258 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:14,258 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:14,258 DEBUG [Thread-799] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:54:14,258 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:14,258 DEBUG [Thread-799] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:54:14,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:14,259 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:14,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:14,269 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=98, memsize=47.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/.tmp/A/6b838f77696840df9ac1aafb538daec5 2023-04-25 02:54:14,288 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=98 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/.tmp/B/e2f4d75feec94c57b6eb0c32ef4fb5fa 2023-04-25 02:54:14,358 DEBUG [Thread-799] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:54:14,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:54:14,362 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:54:14,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:54:14,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:54:14,366 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:54:14,366 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:14,366 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:54:14,366 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:54:14,367 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:14,367 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:14,368 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:14,368 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:14,368 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:14,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:14,369 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:14,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:14,369 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:54:14,369 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:14,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:54:14,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:14,369 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:54:14,370 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:54:14,370 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:54:14,370 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:54:14,370 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:14,370 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:54:14,370 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:54:14,370 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:54:14,370 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. 2023-04-25 02:54:14,370 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. started... 2023-04-25 02:54:14,370 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. as already flushing 2023-04-25 02:54:14,370 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. 2023-04-25 02:54:14,371 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:54:14,371 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:54:14,371 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:14,371 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:54:14,371 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:54:14,376 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:14,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:14,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:14,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:14,377 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:14,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:14,377 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:14,377 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:54:14,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:14,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:14,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:14,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:14,378 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:54:14,378 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6e1331ca[Count = 0] remaining members to acquire global barrier 2023-04-25 02:54:14,378 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:54:14,379 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:14,380 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:14,380 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:14,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:14,380 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:54:14,380 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:54:14,380 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:54:14,380 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:14,380 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:54:14,382 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:54:14,382 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:14,382 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:14,382 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:54:14,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:14,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:14,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:14,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:14,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:14,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:14,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:14,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:14,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:14,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:14,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:54:14,385 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:54:14,385 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:54:14,385 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:54:14,385 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:54:14,385 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:54:14,385 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:54:14,387 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:14,387 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:14,387 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:14,387 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:14,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:14,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:14,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:14,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:14,387 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:14,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:14,388 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:14,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:14,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:14,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:14,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:14,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:14,390 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:14,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:14,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:14,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:14,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:14,393 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:14,393 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:14,393 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:14,394 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:14,394 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:14,394 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:14,394 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:14,394 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:14,394 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:54:14,394 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:54:14,394 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:14,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:14,394 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:14,394 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:14,394 DEBUG [Thread-799] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:54:14,394 DEBUG [Thread-799] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:54:14,394 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:54:14,395 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:14,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:14,402 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:14,402 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:14,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53692 deadline: 1682391314398, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:14,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53610 deadline: 1682391314400, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:14,409 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:14,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53664 deadline: 1682391314408, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:14,411 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:14,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53642 deadline: 1682391314410, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:14,412 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:14,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53650 deadline: 1682391314411, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:14,494 DEBUG [Thread-799] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:54:14,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:54:14,497 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:54:14,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:54:14,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:54:14,498 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:54:14,499 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:14,499 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:54:14,499 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:54:14,499 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:14,499 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:14,500 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:14,500 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:14,500 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:14,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:14,501 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:14,501 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:54:14,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:14,524 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:14,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:54:14,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:14,525 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:54:14,536 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:54:14,536 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:54:14,536 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:14,536 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:54:14,537 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:54:14,537 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:54:14,537 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. 2023-04-25 02:54:14,537 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. started... 2023-04-25 02:54:14,537 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. as already flushing 2023-04-25 02:54:14,537 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. 2023-04-25 02:54:14,537 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:54:14,537 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:54:14,537 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:14,537 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:54:14,537 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:54:14,539 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:14,539 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:14,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:14,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:14,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:14,540 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:14,540 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:54:14,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:14,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:14,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:14,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:14,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:14,541 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:54:14,542 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6711fc78[Count = 0] remaining members to acquire global barrier 2023-04-25 02:54:14,542 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:54:14,542 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:14,543 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:14,543 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:14,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:14,543 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:54:14,543 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:54:14,543 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:54:14,543 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:14,543 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:54:14,545 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:14,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:14,545 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:54:14,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:14,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:14,545 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:14,546 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:54:14,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:14,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:14,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:14,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:14,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:14,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:14,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:14,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:54:14,548 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:54:14,548 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:54:14,548 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:54:14,548 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:54:14,548 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:54:14,548 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:54:14,550 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:14,550 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:14,550 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:14,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:14,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:14,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:14,550 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:14,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:14,551 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:14,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:14,551 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:14,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:14,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:14,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:14,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:14,553 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:14,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:14,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:14,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:14,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:14,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:14,555 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:14,555 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:14,555 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:14,555 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:14,555 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:14,555 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:14,555 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:14,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:14,556 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:54:14,556 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:54:14,555 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:14,556 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:14,556 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:14,556 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:14,556 DEBUG [Thread-799] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:54:14,556 DEBUG [Thread-799] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:54:14,556 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:54:14,556 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:14,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:14,656 DEBUG [Thread-799] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:54:14,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:54:14,659 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:54:14,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:54:14,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:54:14,660 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:54:14,660 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:14,660 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:54:14,660 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:54:14,661 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:14,661 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:14,662 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:14,662 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:14,662 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:14,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:14,663 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:14,663 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:54:14,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:14,663 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:14,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:54:14,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:14,663 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:54:14,664 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:54:14,664 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:54:14,664 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:54:14,664 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:14,664 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:54:14,664 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:54:14,664 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:54:14,664 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. 2023-04-25 02:54:14,664 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. started... 2023-04-25 02:54:14,664 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. as already flushing 2023-04-25 02:54:14,664 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. 2023-04-25 02:54:14,664 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:54:14,664 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:54:14,664 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:14,664 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:54:14,664 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:54:14,667 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:14,667 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:14,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:14,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:14,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:14,667 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:14,667 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:54:14,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:14,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:14,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:14,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:14,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:14,669 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:54:14,669 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7fe07baf[Count = 0] remaining members to acquire global barrier 2023-04-25 02:54:14,669 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:54:14,669 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:14,670 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:14,670 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:14,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:14,670 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:54:14,671 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:54:14,671 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:14,671 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:54:14,671 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:54:14,672 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:14,672 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:54:14,672 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:14,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:14,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:14,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:14,672 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:54:14,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:14,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:14,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:14,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:14,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:14,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:14,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:14,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:54:14,675 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:54:14,675 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:54:14,675 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:54:14,675 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:54:14,675 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:54:14,675 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:54:14,683 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:14,683 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:14,683 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:14,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:14,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:14,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:14,683 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:14,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:14,683 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:14,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:14,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:14,684 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:14,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:14,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:14,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:14,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:14,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:14,686 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:14,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:14,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:14,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:14,691 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:14,691 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:14,691 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:14,691 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:54:14,691 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:54:14,691 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:14,691 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:14,691 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:14,691 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:14,691 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:14,691 DEBUG [Thread-799] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:54:14,692 DEBUG [Thread-799] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:54:14,692 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:14,692 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:14,692 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:14,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:14,693 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:14,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:14,708 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=98 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/.tmp/C/6a48af4e5ef1489fb6a59a5bd89957b5 2023-04-25 02:54:14,713 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/.tmp/A/6b838f77696840df9ac1aafb538daec5 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/A/6b838f77696840df9ac1aafb538daec5 2023-04-25 02:54:14,719 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/A/6b838f77696840df9ac1aafb538daec5, entries=150, sequenceid=98, filesize=25.1 K 2023-04-25 02:54:14,720 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/.tmp/B/e2f4d75feec94c57b6eb0c32ef4fb5fa as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/B/e2f4d75feec94c57b6eb0c32ef4fb5fa 2023-04-25 02:54:14,725 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/B/e2f4d75feec94c57b6eb0c32ef4fb5fa, entries=150, sequenceid=98, filesize=11.6 K 2023-04-25 02:54:14,726 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/.tmp/C/6a48af4e5ef1489fb6a59a5bd89957b5 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/C/6a48af4e5ef1489fb6a59a5bd89957b5 2023-04-25 02:54:14,730 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/C/6a48af4e5ef1489fb6a59a5bd89957b5, entries=150, sequenceid=98, filesize=11.6 K 2023-04-25 02:54:14,731 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~138.65 KB/141980, heapSize ~390.41 KB/399776, currentSize=53.67 KB/54960 for 388bc1373cd34a10ba8be8ddc731e198 in 964ms, sequenceid=98, compaction requested=true 2023-04-25 02:54:14,731 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 388bc1373cd34a10ba8be8ddc731e198: 2023-04-25 02:54:14,731 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 02:54:14,731 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-25 02:54:14,731 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 02:54:14,731 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-25 02:54:14,731 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-25 02:54:14,733 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 102984 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-25 02:54:14,733 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HStore(1912): 388bc1373cd34a10ba8be8ddc731e198/A is initiating minor compaction (all files) 2023-04-25 02:54:14,733 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47544 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-25 02:54:14,733 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 388bc1373cd34a10ba8be8ddc731e198/A in TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. 2023-04-25 02:54:14,733 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HStore(1912): 388bc1373cd34a10ba8be8ddc731e198/B is initiating minor compaction (all files) 2023-04-25 02:54:14,733 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/A/526ce0eef8a04f3a8c1b08316b8dc391, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/A/25246c3480d543cea2f7d1c2b9eb48f8, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/A/7c2d5bcd4cbd44eb9aa87771ea9ec9ac, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/A/6b838f77696840df9ac1aafb538daec5] into tmpdir=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/.tmp, totalSize=100.6 K 2023-04-25 02:54:14,733 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 388bc1373cd34a10ba8be8ddc731e198/B in TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. 2023-04-25 02:54:14,733 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/B/9d672d5b816b4f128fd34506fb1fa281, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/B/e411495647fd4bfe994149ae41b61d23, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/B/2500b86ed0524a3488809f7eb6811311, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/B/e2f4d75feec94c57b6eb0c32ef4fb5fa] into tmpdir=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/.tmp, totalSize=46.4 K 2023-04-25 02:54:14,733 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting 526ce0eef8a04f3a8c1b08316b8dc391, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=15, earliestPutTs=1682391252108 2023-04-25 02:54:14,734 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting 9d672d5b816b4f128fd34506fb1fa281, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=15, earliestPutTs=1682391252108 2023-04-25 02:54:14,734 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting 25246c3480d543cea2f7d1c2b9eb48f8, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1682391252160 2023-04-25 02:54:14,734 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting e411495647fd4bfe994149ae41b61d23, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1682391252160 2023-04-25 02:54:14,734 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting 7c2d5bcd4cbd44eb9aa87771ea9ec9ac, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=62, earliestPutTs=1682391252482 2023-04-25 02:54:14,734 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting 2500b86ed0524a3488809f7eb6811311, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=62, earliestPutTs=1682391252482 2023-04-25 02:54:14,735 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting 6b838f77696840df9ac1aafb538daec5, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=98, earliestPutTs=1682391253143 2023-04-25 02:54:14,735 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting e2f4d75feec94c57b6eb0c32ef4fb5fa, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=98, earliestPutTs=1682391253143 2023-04-25 02:54:14,758 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] throttle.PressureAwareThroughputController(145): 388bc1373cd34a10ba8be8ddc731e198#B#compaction#111 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-25 02:54:14,779 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] throttle.PressureAwareThroughputController(145): 388bc1373cd34a10ba8be8ddc731e198#A#compaction#112 average throughput is 1.31 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-25 02:54:14,792 DEBUG [Thread-799] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:54:14,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:54:14,797 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:54:14,798 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/.tmp/B/ad7625f91b8645ad838990f1acc5878e as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/B/ad7625f91b8645ad838990f1acc5878e 2023-04-25 02:54:14,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:54:14,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:54:14,806 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:54:14,806 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:14,806 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:54:14,806 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:54:14,807 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:14,807 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:14,806 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 388bc1373cd34a10ba8be8ddc731e198/B of 388bc1373cd34a10ba8be8ddc731e198 into ad7625f91b8645ad838990f1acc5878e(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 02:54:14,808 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 388bc1373cd34a10ba8be8ddc731e198: 2023-04-25 02:54:14,808 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198., storeName=388bc1373cd34a10ba8be8ddc731e198/B, priority=12, startTime=1682391254731; duration=0sec 2023-04-25 02:54:14,808 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 02:54:14,808 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-25 02:54:14,809 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:14,809 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:14,809 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:14,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:14,809 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:14,809 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:54:14,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:14,810 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:14,810 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47544 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-25 02:54:14,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:54:14,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:14,810 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HStore(1912): 388bc1373cd34a10ba8be8ddc731e198/C is initiating minor compaction (all files) 2023-04-25 02:54:14,810 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:54:14,810 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 388bc1373cd34a10ba8be8ddc731e198/C in TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. 2023-04-25 02:54:14,810 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/C/ea45787746b740a88796fac1b681bb91, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/C/d0dcc153e64a47cd9669afca16d2538f, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/C/d4d62a806d6a48a380c0e88b0525fca1, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/C/6a48af4e5ef1489fb6a59a5bd89957b5] into tmpdir=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/.tmp, totalSize=46.4 K 2023-04-25 02:54:14,811 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting ea45787746b740a88796fac1b681bb91, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=15, earliestPutTs=1682391252108 2023-04-25 02:54:14,812 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:54:14,812 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting d0dcc153e64a47cd9669afca16d2538f, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1682391252160 2023-04-25 02:54:14,812 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:54:14,812 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:54:14,812 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:14,812 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:54:14,812 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:54:14,812 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:54:14,812 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting d4d62a806d6a48a380c0e88b0525fca1, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=62, earliestPutTs=1682391252482 2023-04-25 02:54:14,813 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. 2023-04-25 02:54:14,813 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. started... 2023-04-25 02:54:14,813 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 388bc1373cd34a10ba8be8ddc731e198 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-25 02:54:14,813 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 388bc1373cd34a10ba8be8ddc731e198, store=A 2023-04-25 02:54:14,813 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:54:14,813 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 388bc1373cd34a10ba8be8ddc731e198, store=B 2023-04-25 02:54:14,813 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:54:14,813 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 388bc1373cd34a10ba8be8ddc731e198, store=C 2023-04-25 02:54:14,813 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:54:14,814 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting 6a48af4e5ef1489fb6a59a5bd89957b5, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=98, earliestPutTs=1682391253143 2023-04-25 02:54:14,832 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] throttle.PressureAwareThroughputController(145): 388bc1373cd34a10ba8be8ddc731e198#C#compaction#114 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-25 02:54:14,864 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023042522672a3f84f845868d98a863ca671072 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042522672a3f84f845868d98a863ca671072 2023-04-25 02:54:14,875 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/.tmp/C/3106d5b1f8134a99ad4d07af64836bd8 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/C/3106d5b1f8134a99ad4d07af64836bd8 2023-04-25 02:54:14,888 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=110, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/.tmp/A/4a74096ea9a449fd99fa95ee4c6f0e9b 2023-04-25 02:54:14,888 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 388bc1373cd34a10ba8be8ddc731e198/C of 388bc1373cd34a10ba8be8ddc731e198 into 3106d5b1f8134a99ad4d07af64836bd8(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 02:54:14,889 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 388bc1373cd34a10ba8be8ddc731e198: 2023-04-25 02:54:14,889 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198., storeName=388bc1373cd34a10ba8be8ddc731e198/C, priority=12, startTime=1682391254731; duration=0sec 2023-04-25 02:54:14,889 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 02:54:14,908 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. as already flushing 2023-04-25 02:54:14,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(9158): Flush requested on 388bc1373cd34a10ba8be8ddc731e198 2023-04-25 02:54:14,987 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682391254956, max timestamp=1682391254979Num uniques 859; 2023-04-25 02:54:14,987 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682391254956, max timestamp=1682391254981Num uniques 859; 2023-04-25 02:54:14,988 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-25 02:54:14,988 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-25 02:54:14,989 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1682391254961, max timestamp=1682391254979Num uniques 859; 2023-04-25 02:54:14,989 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-04-25 02:54:15,033 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:15,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53610 deadline: 1682391315027, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:15,034 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:15,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53692 deadline: 1682391315029, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:15,034 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:15,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53642 deadline: 1682391315031, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:15,043 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:15,043 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:15,044 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53650 deadline: 1682391315033, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:15,044 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53664 deadline: 1682391315034, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:15,135 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:15,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53610 deadline: 1682391315135, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:15,146 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:15,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53642 deadline: 1682391315138, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:15,146 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:15,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53692 deadline: 1682391315138, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:15,146 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:15,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53664 deadline: 1682391315145, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:15,147 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:15,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53650 deadline: 1682391315146, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:15,300 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/.tmp/A/849a15cde2b844a38357800035099e73 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/A/849a15cde2b844a38357800035099e73 2023-04-25 02:54:15,307 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 388bc1373cd34a10ba8be8ddc731e198/A of 388bc1373cd34a10ba8be8ddc731e198 into 849a15cde2b844a38357800035099e73(size=25.3 K), total size for store is 25.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 02:54:15,307 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 388bc1373cd34a10ba8be8ddc731e198: 2023-04-25 02:54:15,307 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198., storeName=388bc1373cd34a10ba8be8ddc731e198/A, priority=12, startTime=1682391254731; duration=0sec 2023-04-25 02:54:15,307 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 02:54:15,341 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:15,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53610 deadline: 1682391315338, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:15,342 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=110 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/.tmp/B/8b574ddc194a4d29b92fcf273e5ce506 2023-04-25 02:54:15,350 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:15,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53650 deadline: 1682391315348, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:15,350 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:15,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53642 deadline: 1682391315348, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:15,350 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:15,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53692 deadline: 1682391315348, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:15,351 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:15,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53664 deadline: 1682391315349, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:15,383 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=110 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/.tmp/C/a961f1057d05493a80e87dc1f76b1fdc 2023-04-25 02:54:15,393 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/.tmp/A/4a74096ea9a449fd99fa95ee4c6f0e9b as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/A/4a74096ea9a449fd99fa95ee4c6f0e9b 2023-04-25 02:54:15,403 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/A/4a74096ea9a449fd99fa95ee4c6f0e9b, entries=150, sequenceid=110, filesize=25.1 K 2023-04-25 02:54:15,404 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/.tmp/B/8b574ddc194a4d29b92fcf273e5ce506 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/B/8b574ddc194a4d29b92fcf273e5ce506 2023-04-25 02:54:15,409 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/B/8b574ddc194a4d29b92fcf273e5ce506, entries=150, sequenceid=110, filesize=11.6 K 2023-04-25 02:54:15,410 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/.tmp/C/a961f1057d05493a80e87dc1f76b1fdc as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/C/a961f1057d05493a80e87dc1f76b1fdc 2023-04-25 02:54:15,417 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/C/a961f1057d05493a80e87dc1f76b1fdc, entries=150, sequenceid=110, filesize=11.6 K 2023-04-25 02:54:15,417 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=131.55 KB/134705 for 388bc1373cd34a10ba8be8ddc731e198 in 604ms, sequenceid=110, compaction requested=false 2023-04-25 02:54:15,418 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 388bc1373cd34a10ba8be8ddc731e198: 2023-04-25 02:54:15,418 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. 2023-04-25 02:54:15,418 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:54:15,418 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:54:15,418 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:15,418 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:54:15,418 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:54:15,420 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:15,420 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:15,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:15,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:15,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:15,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:15,420 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:15,420 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:54:15,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:15,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:15,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:15,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:15,422 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:54:15,422 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5e57c925[Count = 0] remaining members to acquire global barrier 2023-04-25 02:54:15,422 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:54:15,422 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:15,423 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:15,423 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:15,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:15,423 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:54:15,423 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:54:15,423 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:54:15,426 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:15,426 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:54:15,431 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:15,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:15,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:15,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:15,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:15,432 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:54:15,432 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:15,432 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:54:15,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:15,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:15,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:15,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:15,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:15,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:15,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:54:15,434 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:54:15,434 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:54:15,434 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:54:15,434 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:54:15,434 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:54:15,434 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:54:15,436 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:15,436 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:15,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:15,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:15,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:15,436 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:15,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:15,436 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:15,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:15,436 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:15,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:15,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:15,437 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:15,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:15,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:15,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:15,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:15,441 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:15,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:15,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:15,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:15,443 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:15,443 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:15,443 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:15,443 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:15,443 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:15,443 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:54:15,443 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:54:15,443 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:15,443 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:15,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:15,443 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:15,444 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:15,444 DEBUG [Thread-799] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:54:15,444 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:15,444 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:15,444 DEBUG [Thread-799] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:54:15,444 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:15,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:15,544 DEBUG [Thread-799] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:54:15,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:54:15,546 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:54:15,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:54:15,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:54:15,547 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:54:15,547 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:15,547 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:54:15,547 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:54:15,548 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:15,548 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:15,549 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:15,549 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:15,549 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:15,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:15,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:15,549 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:15,550 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:54:15,550 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:15,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:54:15,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:15,550 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:54:15,550 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:54:15,550 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:54:15,551 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:54:15,551 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:15,551 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:54:15,551 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:54:15,551 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:54:15,551 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. 2023-04-25 02:54:15,551 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. started... 2023-04-25 02:54:15,551 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 388bc1373cd34a10ba8be8ddc731e198 3/3 column families, dataSize=131.55 KB heapSize=374.30 KB 2023-04-25 02:54:15,551 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 388bc1373cd34a10ba8be8ddc731e198, store=A 2023-04-25 02:54:15,551 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 02:54:15,551 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 388bc1373cd34a10ba8be8ddc731e198, store=B 2023-04-25 02:54:15,551 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 02:54:15,551 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 388bc1373cd34a10ba8be8ddc731e198, store=C 2023-04-25 02:54:15,552 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 02:54:15,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(9158): Flush requested on 388bc1373cd34a10ba8be8ddc731e198 2023-04-25 02:54:15,646 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. as already flushing 2023-04-25 02:54:15,661 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:15,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53650 deadline: 1682391315657, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:15,663 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:15,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53642 deadline: 1682391315660, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:15,663 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:15,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53664 deadline: 1682391315660, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:15,664 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:15,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53610 deadline: 1682391315661, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:15,664 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:15,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53692 deadline: 1682391315662, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:15,766 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:15,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53610 deadline: 1682391315765, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:15,767 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:15,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53692 deadline: 1682391315765, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:15,767 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:15,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53664 deadline: 1682391315765, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:15,969 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023042509234cdf52b440c8a1178dc2e3476b75 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042509234cdf52b440c8a1178dc2e3476b75 2023-04-25 02:54:15,972 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:15,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53692 deadline: 1682391315969, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:15,973 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:15,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53664 deadline: 1682391315970, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:15,973 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:15,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53610 deadline: 1682391315970, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:15,975 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=148, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/.tmp/A/71e2cc2867584dd4900be2b96d734d12 2023-04-25 02:54:16,002 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=148 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/.tmp/B/67f93a6b068a4cde974592dae873f4b3 2023-04-25 02:54:16,163 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:16,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53650 deadline: 1682391316162, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:16,167 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:16,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53642 deadline: 1682391316165, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:16,278 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:16,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53692 deadline: 1682391316276, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:16,278 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:16,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53664 deadline: 1682391316276, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:16,278 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:16,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53610 deadline: 1682391316276, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:16,416 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.09 KB at sequenceid=148 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/.tmp/C/9b1b72180ef444c7aca5c54bc9105341 2023-04-25 02:54:16,422 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/.tmp/A/71e2cc2867584dd4900be2b96d734d12 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/A/71e2cc2867584dd4900be2b96d734d12 2023-04-25 02:54:16,429 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/A/71e2cc2867584dd4900be2b96d734d12, entries=150, sequenceid=148, filesize=25.3 K 2023-04-25 02:54:16,430 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/.tmp/B/67f93a6b068a4cde974592dae873f4b3 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/B/67f93a6b068a4cde974592dae873f4b3 2023-04-25 02:54:16,437 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/B/67f93a6b068a4cde974592dae873f4b3, entries=150, sequenceid=148, filesize=11.8 K 2023-04-25 02:54:16,437 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.StoreScanner(961): StoreScanner already closing. There is no need to updateReaders 2023-04-25 02:54:16,438 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/.tmp/C/9b1b72180ef444c7aca5c54bc9105341 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/C/9b1b72180ef444c7aca5c54bc9105341 2023-04-25 02:54:16,444 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/C/9b1b72180ef444c7aca5c54bc9105341, entries=150, sequenceid=148, filesize=11.8 K 2023-04-25 02:54:16,445 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~131.55 KB/134705, heapSize ~374.26 KB/383240, currentSize=53.67 KB/54960 for 388bc1373cd34a10ba8be8ddc731e198 in 894ms, sequenceid=148, compaction requested=true 2023-04-25 02:54:16,446 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 388bc1373cd34a10ba8be8ddc731e198: 2023-04-25 02:54:16,446 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. 2023-04-25 02:54:16,446 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:54:16,446 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:54:16,446 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:16,446 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:54:16,446 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:54:16,448 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:16,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:16,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:16,449 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:16,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:16,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:16,449 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:16,449 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:54:16,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:16,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:16,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:16,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:16,451 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:54:16,451 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@293d2e44[Count = 0] remaining members to acquire global barrier 2023-04-25 02:54:16,451 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:54:16,451 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:16,452 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:16,452 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:16,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:16,452 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:54:16,452 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:54:16,452 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:54:16,453 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:16,453 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:54:16,454 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:16,454 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:54:16,454 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:16,454 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:54:16,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:16,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:16,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:16,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:16,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:16,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:16,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:16,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:16,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:16,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:16,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:54:16,457 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:54:16,457 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:54:16,457 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:54:16,457 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:54:16,457 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:54:16,457 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:54:16,459 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:16,459 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:16,459 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:16,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:16,459 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:16,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:16,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:16,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:16,459 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:16,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:16,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:16,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:16,460 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:16,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:16,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:16,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:16,461 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:16,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:16,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:16,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:16,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:16,465 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:16,465 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:16,465 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:16,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:16,465 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:16,465 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:16,465 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:16,465 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:16,465 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:16,465 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:16,465 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:16,465 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:16,466 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:54:16,466 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:54:16,466 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:16,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:16,466 DEBUG [Thread-799] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:54:16,466 DEBUG [Thread-799] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:54:16,466 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:54:16,566 DEBUG [Thread-799] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:54:16,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:54:16,569 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:54:16,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:54:16,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:54:16,572 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:54:16,572 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:16,572 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:54:16,572 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:54:16,572 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:16,573 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:16,574 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:16,574 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:16,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:16,574 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:16,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:16,575 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:16,575 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:54:16,575 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:16,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:54:16,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:16,576 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:54:16,576 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:54:16,577 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:54:16,577 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:54:16,577 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:16,577 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:54:16,577 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:54:16,577 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:54:16,577 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. 2023-04-25 02:54:16,577 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. started... 2023-04-25 02:54:16,577 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 388bc1373cd34a10ba8be8ddc731e198 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-25 02:54:16,578 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 388bc1373cd34a10ba8be8ddc731e198, store=A 2023-04-25 02:54:16,578 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:54:16,578 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 388bc1373cd34a10ba8be8ddc731e198, store=B 2023-04-25 02:54:16,578 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:54:16,578 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 388bc1373cd34a10ba8be8ddc731e198, store=C 2023-04-25 02:54:16,578 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:54:16,600 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304251d7fe7daff0042cfbdfe883e02aad585 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304251d7fe7daff0042cfbdfe883e02aad585 2023-04-25 02:54:16,606 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=159, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/.tmp/A/c26090aab11042a7a3676ac917f3de43 2023-04-25 02:54:16,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(9158): Flush requested on 388bc1373cd34a10ba8be8ddc731e198 2023-04-25 02:54:16,782 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. as already flushing 2023-04-25 02:54:16,808 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:16,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53664 deadline: 1682391316806, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:16,809 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=459, cellSize=21015, totalHeapSize=47976, min timestamp=1682391256791, max timestamp=1682391256807Num uniques 859; 2023-04-25 02:54:16,809 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:16,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53692 deadline: 1682391316807, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:16,809 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=21015, suffix heap size=117064,new segment heap size=47976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=459 2023-04-25 02:54:16,809 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=459, cellSize=21015, totalHeapSize=47976, min timestamp=1682391256791, max timestamp=1682391256807Num uniques 859; 2023-04-25 02:54:16,809 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=21015, suffix heap size=117064,new segment heap size=47976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=459 2023-04-25 02:54:16,809 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=459, cellSize=21015, totalHeapSize=47976, min timestamp=1682391256791, max timestamp=1682391256807Num uniques 859; 2023-04-25 02:54:16,809 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=21015, suffix heap size=117064,new segment heap size=47976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=459 2023-04-25 02:54:16,845 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:16,845 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53610 deadline: 1682391316844, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:16,910 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:16,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53664 deadline: 1682391316910, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:16,918 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:16,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53692 deadline: 1682391316912, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:16,947 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:16,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53610 deadline: 1682391316947, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:17,020 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=159 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/.tmp/B/bcce52a27cb64fca8262217a8ec2c06e 2023-04-25 02:54:17,114 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:17,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53664 deadline: 1682391317113, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:17,123 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:17,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53692 deadline: 1682391317120, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:17,149 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:17,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53610 deadline: 1682391317148, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:17,168 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:17,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53650 deadline: 1682391317166, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:17,179 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:17,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53642 deadline: 1682391317177, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:17,416 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:17,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53664 deadline: 1682391317415, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:17,424 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:17,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53692 deadline: 1682391317424, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:17,432 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=159 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/.tmp/C/182dc9262d114df490a115aefff1c753 2023-04-25 02:54:17,438 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/.tmp/A/c26090aab11042a7a3676ac917f3de43 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/A/c26090aab11042a7a3676ac917f3de43 2023-04-25 02:54:17,443 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/A/c26090aab11042a7a3676ac917f3de43, entries=150, sequenceid=159, filesize=25.3 K 2023-04-25 02:54:17,444 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/.tmp/B/bcce52a27cb64fca8262217a8ec2c06e as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/B/bcce52a27cb64fca8262217a8ec2c06e 2023-04-25 02:54:17,449 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/B/bcce52a27cb64fca8262217a8ec2c06e, entries=150, sequenceid=159, filesize=11.8 K 2023-04-25 02:54:17,450 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/.tmp/C/182dc9262d114df490a115aefff1c753 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/C/182dc9262d114df490a115aefff1c753 2023-04-25 02:54:17,454 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/C/182dc9262d114df490a115aefff1c753, entries=150, sequenceid=159, filesize=11.8 K 2023-04-25 02:54:17,454 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:17,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53610 deadline: 1682391317450, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:17,455 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=134.18 KB/137400 for 388bc1373cd34a10ba8be8ddc731e198 in 878ms, sequenceid=159, compaction requested=true 2023-04-25 02:54:17,455 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 388bc1373cd34a10ba8be8ddc731e198: 2023-04-25 02:54:17,455 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. 2023-04-25 02:54:17,455 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:54:17,455 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:54:17,455 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:17,455 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:54:17,455 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:54:17,457 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:17,457 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:17,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:17,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:17,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:17,457 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:17,457 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:54:17,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:17,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:17,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:17,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:17,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:17,459 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:54:17,459 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@41a7e527[Count = 0] remaining members to acquire global barrier 2023-04-25 02:54:17,459 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:54:17,459 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:17,460 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:17,460 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:17,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:17,460 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:54:17,460 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:54:17,460 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:17,461 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:54:17,460 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:54:17,462 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:17,463 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:54:17,463 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:17,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:17,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:17,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:17,463 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:54:17,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:17,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:17,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:17,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:17,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:17,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:17,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:17,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:54:17,466 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:54:17,466 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:54:17,466 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:54:17,466 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:54:17,466 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:54:17,466 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:54:17,468 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:17,468 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:17,468 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:17,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:17,469 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:17,469 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:17,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:17,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:17,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:17,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:17,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:17,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:17,470 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:17,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:17,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:17,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:17,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:17,472 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:17,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:17,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:17,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:17,474 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:17,474 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:17,474 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:17,474 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:17,475 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:54:17,475 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:54:17,475 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:17,475 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:17,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:17,475 DEBUG [Thread-799] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:54:17,475 DEBUG [Thread-799] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:54:17,475 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:17,475 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:17,475 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:17,475 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:17,475 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:17,475 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:17,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:17,575 DEBUG [Thread-799] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:54:17,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:54:17,577 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:54:17,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:54:17,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:54:17,581 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:54:17,581 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:17,581 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:54:17,581 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:54:17,582 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:17,582 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:17,584 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:17,584 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:17,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:17,584 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:17,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:17,584 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:17,584 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:54:17,585 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:17,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:54:17,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:17,585 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:54:17,590 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:54:17,590 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:54:17,590 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:54:17,590 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:17,590 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:54:17,590 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:54:17,591 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:54:17,591 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. 2023-04-25 02:54:17,591 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. started... 2023-04-25 02:54:17,591 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 388bc1373cd34a10ba8be8ddc731e198 3/3 column families, dataSize=134.18 KB heapSize=356.86 KB 2023-04-25 02:54:17,591 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 388bc1373cd34a10ba8be8ddc731e198, store=A 2023-04-25 02:54:17,591 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 02:54:17,591 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 388bc1373cd34a10ba8be8ddc731e198, store=B 2023-04-25 02:54:17,591 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 02:54:17,591 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 388bc1373cd34a10ba8be8ddc731e198, store=C 2023-04-25 02:54:17,592 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 02:54:17,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(9158): Flush requested on 388bc1373cd34a10ba8be8ddc731e198 2023-04-25 02:54:17,921 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. as already flushing 2023-04-25 02:54:17,939 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:17,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53664 deadline: 1682391317937, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:17,939 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:17,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53692 deadline: 1682391317938, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:17,959 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:17,959 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53610 deadline: 1682391317958, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:18,013 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230425472c1c0fc9af423f9ad49363b35bbb5d to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425472c1c0fc9af423f9ad49363b35bbb5d 2023-04-25 02:54:18,043 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:18,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53664 deadline: 1682391318040, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:18,043 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:18,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53692 deadline: 1682391318040, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:18,246 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:18,246 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:18,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53664 deadline: 1682391318245, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:18,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53692 deadline: 1682391318245, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:18,420 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=190, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/.tmp/A/aca01b8fefaf426083704570f714eb1a 2023-04-25 02:54:18,439 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=190 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/.tmp/B/c40efdd9c0884675ba292c302497098f 2023-04-25 02:54:18,549 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:18,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53664 deadline: 1682391318548, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:18,549 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:18,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53692 deadline: 1682391318548, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:18,860 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=190 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/.tmp/C/b929f736a8b24d74be77fe058d07cf43 2023-04-25 02:54:18,866 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/.tmp/A/aca01b8fefaf426083704570f714eb1a as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/A/aca01b8fefaf426083704570f714eb1a 2023-04-25 02:54:18,870 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/A/aca01b8fefaf426083704570f714eb1a, entries=150, sequenceid=190, filesize=25.3 K 2023-04-25 02:54:18,871 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/.tmp/B/c40efdd9c0884675ba292c302497098f as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/B/c40efdd9c0884675ba292c302497098f 2023-04-25 02:54:18,876 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/B/c40efdd9c0884675ba292c302497098f, entries=150, sequenceid=190, filesize=11.8 K 2023-04-25 02:54:18,877 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/.tmp/C/b929f736a8b24d74be77fe058d07cf43 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/C/b929f736a8b24d74be77fe058d07cf43 2023-04-25 02:54:18,881 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/C/b929f736a8b24d74be77fe058d07cf43, entries=150, sequenceid=190, filesize=11.8 K 2023-04-25 02:54:18,882 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~356.81 KB/365376, currentSize=53.67 KB/54960 for 388bc1373cd34a10ba8be8ddc731e198 in 1291ms, sequenceid=190, compaction requested=true 2023-04-25 02:54:18,882 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 388bc1373cd34a10ba8be8ddc731e198: 2023-04-25 02:54:18,882 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. 2023-04-25 02:54:18,882 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:54:18,882 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:54:18,882 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:18,882 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:54:18,882 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:54:18,884 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:18,885 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:18,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:18,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:18,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:18,885 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:18,885 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:54:18,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:18,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:18,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:18,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:18,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:18,886 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:54:18,886 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3de6d3bf[Count = 0] remaining members to acquire global barrier 2023-04-25 02:54:18,886 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:54:18,886 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:18,888 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:18,888 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:18,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:18,888 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:54:18,888 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:54:18,888 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:54:18,888 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:18,888 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:54:18,889 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:54:18,889 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:18,890 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:18,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:18,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:18,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:18,890 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:54:18,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:18,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:18,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:18,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:18,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:18,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:18,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:18,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:54:18,892 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:54:18,892 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:54:18,892 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:54:18,892 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:54:18,892 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:54:18,892 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:54:18,898 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:18,898 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:18,898 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:18,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:18,898 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:18,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:18,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:18,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:18,898 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:18,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:18,899 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:18,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:18,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:18,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:18,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:18,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:18,900 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:18,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:18,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:18,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:18,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:18,904 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:18,904 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:18,904 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:18,904 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:18,905 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:18,904 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:18,905 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:54:18,905 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:54:18,905 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:18,905 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:54:18,905 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:18,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:18,905 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:18,905 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:18,905 DEBUG [Thread-799] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:54:18,905 DEBUG [Thread-799] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:54:18,905 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:18,905 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:18,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:18,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(9158): Flush requested on 388bc1373cd34a10ba8be8ddc731e198 2023-04-25 02:54:18,968 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 388bc1373cd34a10ba8be8ddc731e198 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-25 02:54:18,968 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 388bc1373cd34a10ba8be8ddc731e198, store=A 2023-04-25 02:54:18,968 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:54:18,968 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 388bc1373cd34a10ba8be8ddc731e198, store=B 2023-04-25 02:54:18,968 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:54:18,968 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 388bc1373cd34a10ba8be8ddc731e198, store=C 2023-04-25 02:54:18,968 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:54:19,005 DEBUG [Thread-799] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:54:19,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:54:19,007 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:54:19,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:54:19,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:54:19,009 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:54:19,009 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:19,009 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:54:19,009 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:54:19,009 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:19,009 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:19,010 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:19,010 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:19,011 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:19,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:19,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:19,011 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:19,011 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:54:19,012 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:19,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:54:19,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:19,012 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:54:19,012 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:54:19,012 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:54:19,012 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:54:19,012 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:19,012 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:54:19,013 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:54:19,013 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:54:19,013 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. 2023-04-25 02:54:19,013 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. started... 2023-04-25 02:54:19,013 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. as already flushing 2023-04-25 02:54:19,013 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. 2023-04-25 02:54:19,013 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:54:19,013 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:54:19,013 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:19,013 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:54:19,013 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:54:19,014 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:19,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:19,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:19,014 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:19,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:19,015 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:19,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:19,015 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:54:19,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:19,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:19,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:19,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:19,016 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:54:19,016 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5d982bc7[Count = 0] remaining members to acquire global barrier 2023-04-25 02:54:19,016 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:54:19,016 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:19,019 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:19,019 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:19,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:19,019 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:54:19,019 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:54:19,019 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:19,019 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:54:19,019 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:54:19,020 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:19,020 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:54:19,020 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:19,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:19,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:19,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:19,020 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:54:19,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:19,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:19,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:19,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:19,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:19,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:19,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:19,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:54:19,022 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:54:19,022 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:54:19,022 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:54:19,022 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:54:19,022 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:54:19,023 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:54:19,024 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:19,024 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:19,024 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:19,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:19,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:19,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:19,024 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:19,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:19,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:19,025 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:19,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:19,025 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:19,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:19,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:19,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:19,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:19,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:19,026 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:19,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:19,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:19,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:19,028 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:19,028 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:19,028 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:19,028 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:54:19,028 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:54:19,028 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:19,028 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:19,028 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:19,028 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:19,028 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:19,028 DEBUG [Thread-799] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:54:19,028 DEBUG [Thread-799] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:54:19,029 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:19,029 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:19,029 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:19,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:19,029 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:19,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:19,038 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:19,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53610 deadline: 1682391319033, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:19,058 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:19,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53664 deadline: 1682391319054, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:19,058 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:19,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53692 deadline: 1682391319054, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:19,128 DEBUG [Thread-799] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:54:19,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:54:19,130 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:54:19,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:54:19,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:54:19,132 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:54:19,132 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:19,132 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:54:19,132 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:54:19,132 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:19,132 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:19,135 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:19,135 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:19,135 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:19,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:19,135 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:19,135 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:54:19,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:19,136 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:19,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:54:19,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:19,136 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:54:19,136 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:54:19,136 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:54:19,136 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:54:19,137 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:19,137 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:54:19,137 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:54:19,137 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:54:19,137 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. 2023-04-25 02:54:19,137 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. started... 2023-04-25 02:54:19,137 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. as already flushing 2023-04-25 02:54:19,137 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. 2023-04-25 02:54:19,137 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:54:19,137 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:54:19,137 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:19,137 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:54:19,137 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:54:19,138 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:19,138 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:19,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:19,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:19,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:19,138 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:19,139 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:54:19,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:19,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:19,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:19,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:19,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:19,140 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:54:19,140 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1d17613e[Count = 0] remaining members to acquire global barrier 2023-04-25 02:54:19,140 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:54:19,140 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:19,140 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:19,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53610 deadline: 1682391319139, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:19,141 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:19,141 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:19,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:19,141 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:54:19,141 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:54:19,141 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:54:19,141 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:19,141 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:54:19,143 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:19,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:19,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:19,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:19,143 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:54:19,143 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:19,143 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:54:19,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:19,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:19,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:19,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:19,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:19,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:19,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:19,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:54:19,145 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:54:19,145 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:54:19,146 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:54:19,146 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:54:19,146 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:54:19,146 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:54:19,148 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:19,148 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:19,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:19,148 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:19,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:19,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:19,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:19,148 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:19,148 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:19,148 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:19,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:19,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:19,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:19,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:19,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:19,150 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:19,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:19,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:19,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:19,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:19,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:19,152 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:19,152 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:19,152 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:19,152 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:19,152 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:19,152 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:19,152 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:19,152 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:19,152 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:19,152 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:19,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:19,152 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:54:19,152 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:54:19,152 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:19,152 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:54:19,153 DEBUG [Thread-799] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:54:19,153 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:19,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:19,153 DEBUG [Thread-799] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:54:19,177 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:19,177 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53650 deadline: 1682391319177, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:19,179 DEBUG [Thread-793] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4145 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198., hostname=jenkins-hbase4.apache.org,46697,1682391220023, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor28.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-25 02:54:19,183 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:19,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53642 deadline: 1682391319180, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:19,185 DEBUG [Thread-789] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4154 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198., hostname=jenkins-hbase4.apache.org,46697,1682391220023, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor28.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-25 02:54:19,253 DEBUG [Thread-799] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:54:19,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:54:19,255 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:54:19,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:54:19,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:54:19,270 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:54:19,270 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:19,270 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:54:19,270 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:54:19,271 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:19,271 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:19,272 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:19,272 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:19,272 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:19,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:19,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:19,273 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:19,273 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:54:19,273 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:19,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:54:19,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:19,273 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:54:19,275 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:54:19,275 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:54:19,275 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:54:19,275 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:19,275 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:54:19,282 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:54:19,282 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:54:19,282 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. 2023-04-25 02:54:19,282 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. started... 2023-04-25 02:54:19,282 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. as already flushing 2023-04-25 02:54:19,282 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. 2023-04-25 02:54:19,282 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:54:19,282 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:54:19,282 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:19,282 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:54:19,282 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:54:19,283 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:19,284 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:19,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:19,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:19,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:19,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:19,285 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:19,285 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:54:19,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:19,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:19,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:19,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:19,286 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:54:19,286 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@63e72166[Count = 0] remaining members to acquire global barrier 2023-04-25 02:54:19,286 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:54:19,286 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:19,289 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:19,289 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:19,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:19,289 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:54:19,289 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:54:19,289 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:54:19,290 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:19,290 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:54:19,290 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:54:19,290 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:19,290 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:54:19,290 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:19,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:19,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:19,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:19,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:19,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:19,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:19,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:19,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:19,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:19,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:19,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:54:19,293 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:54:19,293 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:54:19,293 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:54:19,293 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:54:19,293 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:54:19,293 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:54:19,295 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:19,295 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:19,295 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:19,295 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:19,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:19,295 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:19,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:19,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:19,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:19,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:19,296 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:19,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:19,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:19,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:19,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:19,297 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:19,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:19,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:19,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:19,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:19,301 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:19,301 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:19,301 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:19,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:19,301 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:19,301 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:19,301 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:54:19,301 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:54:19,301 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:19,301 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:19,301 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:19,301 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:19,301 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:19,302 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:19,302 DEBUG [Thread-799] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:54:19,302 DEBUG [Thread-799] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:54:19,302 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:19,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:19,302 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:54:19,346 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:19,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53610 deadline: 1682391319342, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:19,383 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304252d4f1c7ee46447a9965531e6515d5c31 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304252d4f1c7ee46447a9965531e6515d5c31 2023-04-25 02:54:19,402 DEBUG [Thread-799] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:54:19,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:54:19,404 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:54:19,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:54:19,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:54:19,408 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:54:19,408 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:19,408 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:54:19,408 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:54:19,409 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:19,409 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:19,410 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:19,410 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:19,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:19,410 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:19,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:19,411 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:19,411 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:54:19,411 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:19,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:54:19,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:19,411 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:54:19,412 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:54:19,412 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:54:19,412 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:54:19,412 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:19,412 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:54:19,416 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:54:19,416 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:54:19,416 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. 2023-04-25 02:54:19,416 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. started... 2023-04-25 02:54:19,416 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. as already flushing 2023-04-25 02:54:19,416 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. 2023-04-25 02:54:19,416 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:54:19,416 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:54:19,416 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:19,416 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:54:19,416 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:54:19,418 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:19,418 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:19,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:19,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:19,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:19,418 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:19,418 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:54:19,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:19,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:19,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:19,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:19,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:19,420 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:54:19,420 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@68976d8e[Count = 0] remaining members to acquire global barrier 2023-04-25 02:54:19,420 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:54:19,420 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:19,421 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:19,421 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:19,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:19,421 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:54:19,421 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:54:19,421 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:54:19,421 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:19,421 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:54:19,423 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:54:19,423 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:19,423 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:19,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:19,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:19,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:19,423 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:54:19,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:19,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:19,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:19,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:19,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:19,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:19,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:19,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:54:19,425 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:54:19,425 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:54:19,426 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:54:19,426 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:54:19,426 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:54:19,426 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:54:19,427 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:19,427 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:19,427 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:19,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:19,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:19,427 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:19,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:19,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:19,427 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:19,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:19,427 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:19,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:19,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:19,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:19,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:19,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:19,429 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:19,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:19,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:19,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:19,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:19,431 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:19,431 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:19,431 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:19,431 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:19,431 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:19,431 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:19,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:19,431 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:54:19,431 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:54:19,431 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:19,431 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:19,431 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:19,431 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:19,432 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:19,432 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:19,432 DEBUG [Thread-799] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:54:19,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:19,432 DEBUG [Thread-799] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:54:19,532 DEBUG [Thread-799] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:54:19,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:54:19,534 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:54:19,535 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:54:19,535 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:54:19,535 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:54:19,535 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:19,535 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:54:19,536 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:54:19,536 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:19,536 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:19,537 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:19,537 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:19,537 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:19,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:19,537 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:19,538 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:54:19,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:19,538 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:19,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:54:19,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:19,538 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:54:19,539 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:54:19,539 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:54:19,539 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:54:19,539 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:19,539 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:54:19,539 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:54:19,539 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:54:19,539 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. 2023-04-25 02:54:19,539 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. started... 2023-04-25 02:54:19,539 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. as already flushing 2023-04-25 02:54:19,539 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. 2023-04-25 02:54:19,539 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:54:19,539 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:54:19,539 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:19,539 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:54:19,539 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:54:19,542 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:19,542 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:19,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:19,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:19,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:19,542 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:19,542 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:54:19,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:19,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:19,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:19,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:19,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:19,544 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:54:19,544 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@74cda548[Count = 0] remaining members to acquire global barrier 2023-04-25 02:54:19,544 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:54:19,544 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:19,545 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:19,545 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:19,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:19,545 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:54:19,545 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:54:19,545 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:54:19,545 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:19,545 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:54:19,546 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:19,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:19,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:19,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:19,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:19,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:19,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:19,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:19,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:19,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:19,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:19,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:54:19,558 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:54:19,558 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:54:19,558 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:54:19,558 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:54:19,558 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:54:19,558 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:54:19,559 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:19,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:19,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:19,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:19,560 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:19,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:19,560 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:19,560 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:54:19,560 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:19,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:19,560 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:19,560 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:54:19,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:19,561 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:19,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:19,561 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:19,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:19,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:19,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:19,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:19,562 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:19,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:19,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:19,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:19,565 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:19,565 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:19,565 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:19,565 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:19,565 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:54:19,565 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:54:19,565 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:19,565 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:54:19,565 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:19,565 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:19,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:19,566 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:19,566 DEBUG [Thread-799] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:54:19,566 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:19,566 DEBUG [Thread-799] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:54:19,566 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:19,566 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:19,566 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:19,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:19,648 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:19,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53610 deadline: 1682391319648, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:19,666 DEBUG [Thread-799] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:54:19,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:54:19,668 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:54:19,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:54:19,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:54:19,669 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:54:19,669 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:19,669 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:54:19,670 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:54:19,670 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:19,670 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:19,671 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:19,671 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:19,671 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:19,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:19,672 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:19,672 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:54:19,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:19,672 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:19,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:54:19,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:19,672 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:54:19,673 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:54:19,673 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:54:19,673 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:54:19,673 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:19,673 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:54:19,673 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:54:19,673 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:54:19,673 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. 2023-04-25 02:54:19,673 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. started... 2023-04-25 02:54:19,673 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. as already flushing 2023-04-25 02:54:19,673 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. 2023-04-25 02:54:19,673 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:54:19,673 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:54:19,673 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:19,673 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:54:19,674 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:54:19,675 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:19,675 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:19,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:19,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:19,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:19,675 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:19,675 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:54:19,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:19,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:19,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:19,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:19,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:19,677 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:54:19,677 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@52f5247e[Count = 0] remaining members to acquire global barrier 2023-04-25 02:54:19,677 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:54:19,677 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:19,678 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:19,678 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:19,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:19,678 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:54:19,678 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:19,678 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:54:19,678 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:54:19,678 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:54:19,680 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:19,680 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:54:19,680 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:19,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:19,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:19,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:19,680 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:54:19,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:19,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:19,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:19,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:19,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:19,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:19,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:19,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:54:19,683 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:54:19,683 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:54:19,683 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:54:19,683 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:54:19,683 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:54:19,683 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:54:19,684 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:19,684 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:19,684 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:19,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:19,684 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:19,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:19,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:19,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:19,685 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:19,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:19,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:19,685 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:19,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:19,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:19,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:19,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:19,686 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:19,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:19,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:19,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:19,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:19,691 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:19,691 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:19,691 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:19,691 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:19,691 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:19,691 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:19,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:19,691 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:54:19,691 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:54:19,691 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:19,691 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:19,691 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:19,691 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:19,691 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:54:19,691 DEBUG [Thread-799] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:54:19,691 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:19,691 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:19,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:19,692 DEBUG [Thread-799] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:54:19,791 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=202, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/.tmp/A/9e04451bc8684802896548ec0b573174 2023-04-25 02:54:19,792 DEBUG [Thread-799] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:54:19,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:54:19,794 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:54:19,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:54:19,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:54:19,796 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:54:19,796 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:19,796 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:54:19,796 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:54:19,796 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:19,796 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:19,798 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:19,798 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:19,798 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:19,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:19,799 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:19,799 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:54:19,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:19,799 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:19,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:54:19,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:19,800 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:54:19,800 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:54:19,800 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:54:19,800 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:54:19,800 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:19,800 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:54:19,800 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:54:19,800 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:54:19,800 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. 2023-04-25 02:54:19,800 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. started... 2023-04-25 02:54:19,801 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. as already flushing 2023-04-25 02:54:19,801 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. 2023-04-25 02:54:19,801 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:54:19,801 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:54:19,801 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:19,801 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:54:19,801 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:54:19,802 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:19,802 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:19,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:19,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:19,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:19,803 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:19,803 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:54:19,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:19,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:19,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:19,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:19,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:19,806 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:54:19,806 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@8c3d357[Count = 0] remaining members to acquire global barrier 2023-04-25 02:54:19,806 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:54:19,807 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:19,807 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=202 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/.tmp/B/314f2f0378cb41f58618a457bc84285f 2023-04-25 02:54:19,808 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:19,808 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:19,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:19,808 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:54:19,808 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:54:19,808 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:54:19,808 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:19,808 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:54:19,809 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:19,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:19,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:19,809 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:54:19,809 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:19,809 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:54:19,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:19,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:19,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:19,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:19,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:19,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:19,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:19,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:19,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:54:19,813 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:54:19,813 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:54:19,813 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:54:19,813 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:54:19,813 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:54:19,813 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:54:19,816 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:19,816 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:19,816 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:19,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:19,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:19,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:19,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:19,816 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:19,816 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:19,817 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:19,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:19,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:19,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:19,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:19,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:19,820 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=202 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/.tmp/C/17375cdc295943b5a308615ca8fe5afc 2023-04-25 02:54:19,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:19,824 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:19,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:19,826 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/.tmp/A/9e04451bc8684802896548ec0b573174 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/A/9e04451bc8684802896548ec0b573174 2023-04-25 02:54:19,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:19,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:19,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:19,830 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:19,830 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:19,830 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:19,830 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:19,830 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:19,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:19,830 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:19,830 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:19,830 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:19,830 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:19,830 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:19,830 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:19,830 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:54:19,830 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:54:19,830 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:54:19,831 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:19,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:19,831 DEBUG [Thread-799] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:54:19,831 DEBUG [Thread-799] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:54:19,835 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/A/9e04451bc8684802896548ec0b573174, entries=200, sequenceid=202, filesize=32.1 K 2023-04-25 02:54:19,836 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/.tmp/B/314f2f0378cb41f58618a457bc84285f as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/B/314f2f0378cb41f58618a457bc84285f 2023-04-25 02:54:19,842 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/B/314f2f0378cb41f58618a457bc84285f, entries=150, sequenceid=202, filesize=11.8 K 2023-04-25 02:54:19,843 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/.tmp/C/17375cdc295943b5a308615ca8fe5afc as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/C/17375cdc295943b5a308615ca8fe5afc 2023-04-25 02:54:19,848 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/C/17375cdc295943b5a308615ca8fe5afc, entries=150, sequenceid=202, filesize=11.8 K 2023-04-25 02:54:19,848 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 388bc1373cd34a10ba8be8ddc731e198 in 880ms, sequenceid=202, compaction requested=true 2023-04-25 02:54:19,849 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 388bc1373cd34a10ba8be8ddc731e198: 2023-04-25 02:54:19,849 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-25 02:54:19,849 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 02:54:19,849 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 02:54:19,849 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 02:54:19,849 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-25 02:54:19,851 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 162204 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-25 02:54:19,851 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 72054 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-25 02:54:19,851 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HStore(1912): 388bc1373cd34a10ba8be8ddc731e198/A is initiating minor compaction (all files) 2023-04-25 02:54:19,851 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HStore(1912): 388bc1373cd34a10ba8be8ddc731e198/B is initiating minor compaction (all files) 2023-04-25 02:54:19,852 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 388bc1373cd34a10ba8be8ddc731e198/A in TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. 2023-04-25 02:54:19,852 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 388bc1373cd34a10ba8be8ddc731e198/B in TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. 2023-04-25 02:54:19,852 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/A/849a15cde2b844a38357800035099e73, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/A/4a74096ea9a449fd99fa95ee4c6f0e9b, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/A/71e2cc2867584dd4900be2b96d734d12, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/A/c26090aab11042a7a3676ac917f3de43, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/A/aca01b8fefaf426083704570f714eb1a, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/A/9e04451bc8684802896548ec0b573174] into tmpdir=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/.tmp, totalSize=158.4 K 2023-04-25 02:54:19,852 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/B/ad7625f91b8645ad838990f1acc5878e, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/B/8b574ddc194a4d29b92fcf273e5ce506, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/B/67f93a6b068a4cde974592dae873f4b3, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/B/bcce52a27cb64fca8262217a8ec2c06e, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/B/c40efdd9c0884675ba292c302497098f, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/B/314f2f0378cb41f58618a457bc84285f] into tmpdir=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/.tmp, totalSize=70.4 K 2023-04-25 02:54:19,852 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting 849a15cde2b844a38357800035099e73, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=98, earliestPutTs=1682391253143 2023-04-25 02:54:19,853 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting ad7625f91b8645ad838990f1acc5878e, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=98, earliestPutTs=1682391253143 2023-04-25 02:54:19,853 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting 4a74096ea9a449fd99fa95ee4c6f0e9b, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=110, earliestPutTs=1682391253781 2023-04-25 02:54:19,853 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting 8b574ddc194a4d29b92fcf273e5ce506, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=110, earliestPutTs=1682391253781 2023-04-25 02:54:19,854 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting 71e2cc2867584dd4900be2b96d734d12, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=148, earliestPutTs=1682391255027 2023-04-25 02:54:19,854 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting 67f93a6b068a4cde974592dae873f4b3, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=148, earliestPutTs=1682391255027 2023-04-25 02:54:19,855 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting bcce52a27cb64fca8262217a8ec2c06e, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=159, earliestPutTs=1682391255654 2023-04-25 02:54:19,855 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting c26090aab11042a7a3676ac917f3de43, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=159, earliestPutTs=1682391255654 2023-04-25 02:54:19,855 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting c40efdd9c0884675ba292c302497098f, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=190, earliestPutTs=1682391256830 2023-04-25 02:54:19,855 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting aca01b8fefaf426083704570f714eb1a, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=190, earliestPutTs=1682391256830 2023-04-25 02:54:19,856 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting 314f2f0378cb41f58618a457bc84285f, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=202, earliestPutTs=1682391257936 2023-04-25 02:54:19,856 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting 9e04451bc8684802896548ec0b573174, keycount=200, bloomtype=ROW, size=32.1 K, encoding=NONE, compression=NONE, seqNum=202, earliestPutTs=1682391257928 2023-04-25 02:54:19,875 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] throttle.PressureAwareThroughputController(145): 388bc1373cd34a10ba8be8ddc731e198#B#compaction#129 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-25 02:54:19,901 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] throttle.PressureAwareThroughputController(145): 388bc1373cd34a10ba8be8ddc731e198#A#compaction#130 average throughput is 0.94 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-25 02:54:19,909 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/.tmp/B/522cb9c8f84b426181ee9bb7dc9fc27e as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/B/522cb9c8f84b426181ee9bb7dc9fc27e 2023-04-25 02:54:19,916 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 388bc1373cd34a10ba8be8ddc731e198/B of 388bc1373cd34a10ba8be8ddc731e198 into 522cb9c8f84b426181ee9bb7dc9fc27e(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 02:54:19,916 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 388bc1373cd34a10ba8be8ddc731e198: 2023-04-25 02:54:19,916 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198., storeName=388bc1373cd34a10ba8be8ddc731e198/B, priority=10, startTime=1682391259849; duration=0sec 2023-04-25 02:54:19,916 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 02:54:19,916 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-25 02:54:19,918 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 72054 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-25 02:54:19,918 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HStore(1912): 388bc1373cd34a10ba8be8ddc731e198/C is initiating minor compaction (all files) 2023-04-25 02:54:19,918 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 388bc1373cd34a10ba8be8ddc731e198/C in TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. 2023-04-25 02:54:19,918 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/C/3106d5b1f8134a99ad4d07af64836bd8, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/C/a961f1057d05493a80e87dc1f76b1fdc, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/C/9b1b72180ef444c7aca5c54bc9105341, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/C/182dc9262d114df490a115aefff1c753, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/C/b929f736a8b24d74be77fe058d07cf43, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/C/17375cdc295943b5a308615ca8fe5afc] into tmpdir=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/.tmp, totalSize=70.4 K 2023-04-25 02:54:19,919 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting 3106d5b1f8134a99ad4d07af64836bd8, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=98, earliestPutTs=1682391253143 2023-04-25 02:54:19,919 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting a961f1057d05493a80e87dc1f76b1fdc, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=110, earliestPutTs=1682391253781 2023-04-25 02:54:19,920 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting 9b1b72180ef444c7aca5c54bc9105341, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=148, earliestPutTs=1682391255027 2023-04-25 02:54:19,920 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting 182dc9262d114df490a115aefff1c753, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=159, earliestPutTs=1682391255654 2023-04-25 02:54:19,921 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting b929f736a8b24d74be77fe058d07cf43, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=190, earliestPutTs=1682391256830 2023-04-25 02:54:19,921 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting 17375cdc295943b5a308615ca8fe5afc, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=202, earliestPutTs=1682391257936 2023-04-25 02:54:19,931 DEBUG [Thread-799] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:54:19,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:54:19,934 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:54:19,934 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] throttle.PressureAwareThroughputController(145): 388bc1373cd34a10ba8be8ddc731e198#C#compaction#131 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-25 02:54:19,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:54:19,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:54:19,935 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:54:19,935 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:19,935 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:54:19,935 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:54:19,935 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:19,935 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:19,942 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:19,942 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:19,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:19,942 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:19,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:19,943 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:19,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:54:19,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:19,944 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:19,944 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:54:19,944 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:54:19,944 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:54:19,945 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:54:19,945 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:54:19,945 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:19,945 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:54:19,952 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:54:19,952 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:54:19,952 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. 2023-04-25 02:54:19,952 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. started... 2023-04-25 02:54:19,952 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 388bc1373cd34a10ba8be8ddc731e198 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-25 02:54:19,952 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 388bc1373cd34a10ba8be8ddc731e198, store=A 2023-04-25 02:54:19,953 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:54:19,953 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 388bc1373cd34a10ba8be8ddc731e198, store=B 2023-04-25 02:54:19,953 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:54:19,953 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 388bc1373cd34a10ba8be8ddc731e198, store=C 2023-04-25 02:54:19,953 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:54:19,955 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/.tmp/C/6e9b68a842b74489b4d23700a95e60dd as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/C/6e9b68a842b74489b4d23700a95e60dd 2023-04-25 02:54:19,968 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 388bc1373cd34a10ba8be8ddc731e198/C of 388bc1373cd34a10ba8be8ddc731e198 into 6e9b68a842b74489b4d23700a95e60dd(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 02:54:19,968 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 388bc1373cd34a10ba8be8ddc731e198: 2023-04-25 02:54:19,968 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198., storeName=388bc1373cd34a10ba8be8ddc731e198/C, priority=10, startTime=1682391259849; duration=0sec 2023-04-25 02:54:19,968 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 02:54:20,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(9158): Flush requested on 388bc1373cd34a10ba8be8ddc731e198 2023-04-25 02:54:20,064 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. as already flushing 2023-04-25 02:54:20,080 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:20,081 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53664 deadline: 1682391320077, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:20,081 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:20,081 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53692 deadline: 1682391320078, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:20,150 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:20,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53610 deadline: 1682391320149, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:20,182 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:20,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53664 deadline: 1682391320181, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:20,183 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:20,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53692 deadline: 1682391320182, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:20,385 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:20,386 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53664 deadline: 1682391320384, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:20,388 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230425c0c17081c93644cc98a53e99e390dd50 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425c0c17081c93644cc98a53e99e390dd50 2023-04-25 02:54:20,388 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:20,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53692 deadline: 1682391320385, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:20,689 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:20,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53664 deadline: 1682391320688, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:20,697 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:20,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53692 deadline: 1682391320695, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:20,733 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/.tmp/A/2eea0cf801f9417594e3f3f0a6dd3c7e as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/A/2eea0cf801f9417594e3f3f0a6dd3c7e 2023-04-25 02:54:20,738 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 388bc1373cd34a10ba8be8ddc731e198/A of 388bc1373cd34a10ba8be8ddc731e198 into 2eea0cf801f9417594e3f3f0a6dd3c7e(size=25.6 K), total size for store is 25.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 02:54:20,738 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 388bc1373cd34a10ba8be8ddc731e198: 2023-04-25 02:54:20,738 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198., storeName=388bc1373cd34a10ba8be8ddc731e198/A, priority=10, startTime=1682391259849; duration=0sec 2023-04-25 02:54:20,739 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 02:54:20,798 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=223, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/.tmp/A/0f7249ffa7844ca7adae49a2cd57a4f6 2023-04-25 02:54:20,831 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=223 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/.tmp/B/0ab45ceb5c454b62b269dd7d54345b64 2023-04-25 02:54:21,158 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:21,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53610 deadline: 1682391321156, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:21,195 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:21,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53664 deadline: 1682391321194, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:21,201 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:21,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53692 deadline: 1682391321200, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:21,251 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=223 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/.tmp/C/14590354535f4d76854ba539310fa1ec 2023-04-25 02:54:21,257 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/.tmp/A/0f7249ffa7844ca7adae49a2cd57a4f6 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/A/0f7249ffa7844ca7adae49a2cd57a4f6 2023-04-25 02:54:21,263 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/A/0f7249ffa7844ca7adae49a2cd57a4f6, entries=150, sequenceid=223, filesize=25.3 K 2023-04-25 02:54:21,263 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.StoreScanner(953): StoreScanner already has the close lock. There is no need to updateReaders 2023-04-25 02:54:21,264 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/.tmp/B/0ab45ceb5c454b62b269dd7d54345b64 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/B/0ab45ceb5c454b62b269dd7d54345b64 2023-04-25 02:54:21,269 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/B/0ab45ceb5c454b62b269dd7d54345b64, entries=150, sequenceid=223, filesize=11.8 K 2023-04-25 02:54:21,271 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/.tmp/C/14590354535f4d76854ba539310fa1ec as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/C/14590354535f4d76854ba539310fa1ec 2023-04-25 02:54:21,275 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/C/14590354535f4d76854ba539310fa1ec, entries=150, sequenceid=223, filesize=11.8 K 2023-04-25 02:54:21,276 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=67.09 KB/68700 for 388bc1373cd34a10ba8be8ddc731e198 in 1324ms, sequenceid=223, compaction requested=false 2023-04-25 02:54:21,277 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 388bc1373cd34a10ba8be8ddc731e198: 2023-04-25 02:54:21,277 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. 2023-04-25 02:54:21,277 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:54:21,277 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:54:21,277 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:21,277 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:54:21,277 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:54:21,278 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:21,278 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:21,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:21,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:21,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:21,279 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:21,279 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:54:21,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:21,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:21,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:21,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:21,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:21,280 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:54:21,280 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5b6b24dc[Count = 0] remaining members to acquire global barrier 2023-04-25 02:54:21,280 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:54:21,280 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:21,282 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:21,282 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:21,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:21,282 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:54:21,283 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:54:21,283 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:21,283 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:54:21,283 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:54:21,284 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:54:21,284 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:21,284 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:21,284 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:54:21,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:21,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:21,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:21,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:21,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:21,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:21,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:21,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:21,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:21,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:21,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:54:21,287 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:54:21,287 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:54:21,287 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:54:21,287 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:54:21,287 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:54:21,287 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:54:21,290 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:21,290 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:21,290 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:21,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:21,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:21,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:21,290 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:21,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:21,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:21,290 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:21,290 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:21,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:21,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:21,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:21,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:21,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:21,291 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:21,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:21,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:21,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:21,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:21,293 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:21,293 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:21,294 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:21,294 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:21,294 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:21,294 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:54:21,294 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:54:21,294 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:21,294 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:21,294 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:21,294 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:21,294 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:21,294 DEBUG [Thread-799] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:54:21,294 DEBUG [Thread-799] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:54:21,294 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:21,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:21,295 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:21,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:21,394 DEBUG [Thread-799] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:54:21,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:54:21,397 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:54:21,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:54:21,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:54:21,398 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:54:21,398 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:21,398 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:54:21,398 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:54:21,398 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:21,398 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:21,400 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:21,401 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:21,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:21,401 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:21,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:21,401 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:21,401 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:54:21,401 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:21,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:54:21,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:21,402 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:54:21,402 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:54:21,402 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:54:21,402 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:54:21,402 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:21,402 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:54:21,402 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:54:21,402 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:54:21,402 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. 2023-04-25 02:54:21,403 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. started... 2023-04-25 02:54:21,403 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 388bc1373cd34a10ba8be8ddc731e198 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-25 02:54:21,403 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 388bc1373cd34a10ba8be8ddc731e198, store=A 2023-04-25 02:54:21,403 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:54:21,403 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 388bc1373cd34a10ba8be8ddc731e198, store=B 2023-04-25 02:54:21,403 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:54:21,403 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 388bc1373cd34a10ba8be8ddc731e198, store=C 2023-04-25 02:54:21,403 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:54:21,817 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304259eaaeea5f8cd493f891fd5706af01f0a to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304259eaaeea5f8cd493f891fd5706af01f0a 2023-04-25 02:54:21,822 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=238, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/.tmp/A/7081619b390a44a9b79a1aa91b66d7d6 2023-04-25 02:54:21,832 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=238 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/.tmp/B/4c8aee74a42f4267aae1cfc5c4096dc6 2023-04-25 02:54:22,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(9158): Flush requested on 388bc1373cd34a10ba8be8ddc731e198 2023-04-25 02:54:22,204 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. as already flushing 2023-04-25 02:54:22,229 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:22,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53692 deadline: 1682391322225, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:22,229 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:22,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53664 deadline: 1682391322226, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:22,243 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=238 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/.tmp/C/a6645645b71c40658dd3f3c2f72ad5ff 2023-04-25 02:54:22,249 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/.tmp/A/7081619b390a44a9b79a1aa91b66d7d6 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/A/7081619b390a44a9b79a1aa91b66d7d6 2023-04-25 02:54:22,253 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/A/7081619b390a44a9b79a1aa91b66d7d6, entries=150, sequenceid=238, filesize=25.3 K 2023-04-25 02:54:22,254 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/.tmp/B/4c8aee74a42f4267aae1cfc5c4096dc6 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/B/4c8aee74a42f4267aae1cfc5c4096dc6 2023-04-25 02:54:22,258 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/B/4c8aee74a42f4267aae1cfc5c4096dc6, entries=150, sequenceid=238, filesize=11.8 K 2023-04-25 02:54:22,259 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/.tmp/C/a6645645b71c40658dd3f3c2f72ad5ff as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/C/a6645645b71c40658dd3f3c2f72ad5ff 2023-04-25 02:54:22,266 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/C/a6645645b71c40658dd3f3c2f72ad5ff, entries=150, sequenceid=238, filesize=11.8 K 2023-04-25 02:54:22,267 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=114.05 KB/116790 for 388bc1373cd34a10ba8be8ddc731e198 in 864ms, sequenceid=238, compaction requested=true 2023-04-25 02:54:22,267 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 388bc1373cd34a10ba8be8ddc731e198: 2023-04-25 02:54:22,267 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. 2023-04-25 02:54:22,267 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:54:22,267 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:54:22,267 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:22,267 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:54:22,267 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:54:22,269 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:22,269 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:22,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:22,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:22,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:22,269 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:22,270 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:54:22,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:22,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:22,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:22,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:22,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:22,271 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:54:22,271 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@42953d1c[Count = 0] remaining members to acquire global barrier 2023-04-25 02:54:22,271 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:54:22,272 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:22,273 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:22,273 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:22,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:22,273 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:54:22,273 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:22,274 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:54:22,273 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:54:22,274 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:54:22,275 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:54:22,275 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:22,275 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:22,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:22,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:22,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:22,275 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:54:22,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:22,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:22,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:22,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:22,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:22,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:22,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:22,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:54:22,277 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:54:22,278 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:54:22,278 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:54:22,278 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:54:22,278 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:54:22,278 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:54:22,280 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:22,280 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:22,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:22,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:22,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:22,280 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:22,280 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:22,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:22,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:22,281 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:22,281 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:22,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:22,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:22,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:22,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:22,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:22,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:22,282 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:22,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:22,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:22,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:22,284 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:22,284 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:22,284 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:22,284 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:22,284 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:22,284 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:54:22,284 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:54:22,284 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:22,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:22,284 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:54:22,284 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:22,284 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:22,284 DEBUG [Thread-799] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:54:22,285 DEBUG [Thread-799] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:54:22,285 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:22,285 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:22,285 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:22,285 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:22,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:22,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(9158): Flush requested on 388bc1373cd34a10ba8be8ddc731e198 2023-04-25 02:54:22,332 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 388bc1373cd34a10ba8be8ddc731e198 3/3 column families, dataSize=127.47 KB heapSize=379.97 KB 2023-04-25 02:54:22,333 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 388bc1373cd34a10ba8be8ddc731e198, store=A 2023-04-25 02:54:22,333 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 02:54:22,333 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 388bc1373cd34a10ba8be8ddc731e198, store=B 2023-04-25 02:54:22,333 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 02:54:22,333 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 388bc1373cd34a10ba8be8ddc731e198, store=C 2023-04-25 02:54:22,333 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 02:54:22,348 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304252ea2ea1a45194e7db499e2226b1d1efa to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304252ea2ea1a45194e7db499e2226b1d1efa 2023-04-25 02:54:22,348 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:22,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53664 deadline: 1682391322346, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:22,351 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:22,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53692 deadline: 1682391322348, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:22,385 DEBUG [Thread-799] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:54:22,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:54:22,387 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:54:22,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:54:22,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:54:22,388 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:54:22,388 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:22,388 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:54:22,388 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:54:22,388 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:22,388 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:22,389 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:22,390 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:22,390 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:22,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:22,390 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:22,390 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:54:22,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:22,391 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:22,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:54:22,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:22,392 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:54:22,392 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:54:22,392 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:54:22,392 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:54:22,392 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:22,392 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:54:22,392 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:54:22,392 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:54:22,392 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. 2023-04-25 02:54:22,392 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. started... 2023-04-25 02:54:22,392 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. as already flushing 2023-04-25 02:54:22,392 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. 2023-04-25 02:54:22,392 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:54:22,392 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:54:22,392 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:22,392 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:54:22,392 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:54:22,397 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:22,397 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:22,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:22,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:22,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:22,398 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:22,398 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:54:22,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:22,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:22,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:22,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:22,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:22,399 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:54:22,399 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:54:22,399 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@39b827e4[Count = 0] remaining members to acquire global barrier 2023-04-25 02:54:22,399 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:22,400 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:22,400 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:22,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:22,400 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:54:22,400 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:54:22,400 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:54:22,400 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:22,401 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:54:22,402 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:54:22,402 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:22,402 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:22,402 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:54:22,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:22,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:22,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:22,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:22,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:22,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:22,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:22,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:22,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:22,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:22,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:54:22,404 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:54:22,404 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:54:22,404 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:54:22,404 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:54:22,404 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:54:22,404 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:54:22,405 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:22,406 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:22,406 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:22,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:22,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:22,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:22,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:22,406 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:22,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:22,406 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:22,406 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:22,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:22,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:22,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:22,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:22,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:22,407 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:22,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:22,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:22,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:22,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:22,410 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:22,410 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:22,410 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:22,410 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:22,410 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:22,410 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:22,410 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:54:22,411 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:54:22,410 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:22,410 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:22,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:22,411 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:54:22,410 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:22,411 DEBUG [Thread-799] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:54:22,411 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:22,411 DEBUG [Thread-799] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:54:22,411 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:22,411 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:22,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:22,451 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:22,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53664 deadline: 1682391322449, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:22,455 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:22,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53692 deadline: 1682391322452, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:22,511 DEBUG [Thread-799] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:54:22,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:54:22,514 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:54:22,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:54:22,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:54:22,515 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:54:22,515 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:22,515 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:54:22,515 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:54:22,516 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:22,516 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:22,517 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:22,517 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:22,517 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:22,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:22,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:22,517 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:22,518 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:54:22,518 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:22,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:54:22,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:22,518 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:54:22,518 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:54:22,519 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:54:22,519 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:54:22,519 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:22,519 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:54:22,519 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:54:22,519 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:54:22,519 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. 2023-04-25 02:54:22,519 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. started... 2023-04-25 02:54:22,519 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. as already flushing 2023-04-25 02:54:22,519 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. 2023-04-25 02:54:22,519 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:54:22,519 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:54:22,519 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:22,519 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:54:22,519 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:54:22,521 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:22,521 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:22,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:22,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:22,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:22,521 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:22,521 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:54:22,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:22,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:22,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:22,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:22,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:22,522 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:54:22,522 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2f7712ef[Count = 0] remaining members to acquire global barrier 2023-04-25 02:54:22,522 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:54:22,523 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:22,524 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:22,524 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:22,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:22,524 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:54:22,524 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:54:22,524 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:54:22,524 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:22,524 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:54:22,526 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:22,526 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:54:22,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:22,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:22,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:22,526 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:22,526 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:54:22,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:22,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:22,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:22,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:22,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:22,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:22,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:22,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:54:22,529 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:54:22,529 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:54:22,529 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:54:22,529 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:54:22,529 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:54:22,529 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:54:22,530 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:22,530 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:22,530 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:22,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:22,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:22,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:22,530 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:22,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:22,531 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:22,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:22,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:22,531 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:22,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:22,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:22,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:22,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:22,533 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:22,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:22,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:22,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:22,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:22,535 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:22,535 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:22,535 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:22,535 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:54:22,535 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:54:22,535 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:22,535 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:22,535 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:22,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:22,535 DEBUG [Thread-799] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:54:22,535 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:22,535 DEBUG [Thread-799] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:54:22,535 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:54:22,535 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:22,536 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:22,536 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:22,536 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:22,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:22,536 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:22,636 DEBUG [Thread-799] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:54:22,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:54:22,638 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:54:22,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:54:22,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:54:22,639 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:54:22,639 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:22,639 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:54:22,639 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:54:22,639 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:22,639 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:22,641 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:22,641 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:22,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:22,641 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:22,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:22,641 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:22,641 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:54:22,642 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:22,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:54:22,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:22,642 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:54:22,642 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:54:22,642 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:54:22,642 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:54:22,642 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:22,642 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:54:22,643 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:54:22,643 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:54:22,643 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. 2023-04-25 02:54:22,643 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. started... 2023-04-25 02:54:22,643 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. as already flushing 2023-04-25 02:54:22,643 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. 2023-04-25 02:54:22,643 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:54:22,643 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:54:22,643 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:22,643 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:54:22,643 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:54:22,646 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:22,646 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:22,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:22,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:22,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:22,646 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:22,646 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:54:22,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:22,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:22,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:22,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:22,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:22,648 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:54:22,648 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2db88e19[Count = 0] remaining members to acquire global barrier 2023-04-25 02:54:22,648 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:54:22,648 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:22,649 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:22,649 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:22,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:22,649 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:54:22,649 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:54:22,649 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:54:22,649 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:22,649 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:54:22,650 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:22,650 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:54:22,651 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:22,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:22,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:22,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:22,651 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:54:22,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:22,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:22,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:22,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:22,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:22,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:22,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:22,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:54:22,653 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:54:22,653 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:54:22,653 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:54:22,653 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:54:22,653 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:54:22,653 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:54:22,654 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:22,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53664 deadline: 1682391322653, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:22,655 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:22,655 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:22,655 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:22,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:22,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:22,655 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:22,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:22,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:22,655 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:22,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:22,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:22,655 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:22,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:22,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:22,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:22,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:22,657 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:22,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:22,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:22,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:22,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:22,659 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:22,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53692 deadline: 1682391322657, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:22,664 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:22,664 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:22,664 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:22,665 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:22,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:22,665 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:22,665 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:22,665 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:22,665 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:54:22,665 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:54:22,665 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:22,665 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:22,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:22,665 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:22,665 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:22,665 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:22,665 DEBUG [Thread-799] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:54:22,665 DEBUG [Thread-799] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:54:22,764 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=260, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/.tmp/A/f9a9473b993647ab9c4f0bf85254b463 2023-04-25 02:54:22,765 DEBUG [Thread-799] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:54:22,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:54:22,768 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:54:22,769 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:54:22,769 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:54:22,769 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:54:22,769 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:22,769 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:54:22,769 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:54:22,770 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:22,770 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:22,771 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:22,771 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:22,771 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:22,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:22,772 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:22,772 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:54:22,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:22,772 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:22,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:54:22,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:22,773 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:54:22,773 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:54:22,773 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:54:22,773 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:54:22,773 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:22,773 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:54:22,773 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:54:22,773 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:54:22,773 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. 2023-04-25 02:54:22,773 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. started... 2023-04-25 02:54:22,773 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. as already flushing 2023-04-25 02:54:22,773 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. 2023-04-25 02:54:22,773 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:54:22,773 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:54:22,773 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:22,773 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:54:22,773 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:54:22,775 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:22,775 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:22,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:22,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:22,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:22,775 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:22,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:22,775 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:54:22,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:22,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:22,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:22,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:22,777 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:54:22,777 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@46ae2b6f[Count = 0] remaining members to acquire global barrier 2023-04-25 02:54:22,777 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:54:22,777 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:22,779 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:22,779 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:22,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:22,779 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:54:22,779 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:54:22,779 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:54:22,779 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:22,779 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:54:22,783 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:54:22,783 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:22,783 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:22,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:22,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:22,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:22,783 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:54:22,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:22,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:22,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:22,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:22,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:22,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:22,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:22,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:54:22,786 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:54:22,786 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:54:22,786 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:54:22,786 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:54:22,786 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:54:22,786 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:54:22,787 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:22,787 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:22,787 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:22,787 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:22,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:22,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:22,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:22,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:22,788 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:22,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:22,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:22,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:22,788 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:22,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:22,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:22,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:22,789 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:22,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:22,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:22,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:22,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:22,792 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:22,792 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:22,792 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:22,792 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:22,792 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:54:22,792 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:54:22,792 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:22,792 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:22,792 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:22,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:22,792 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:22,792 DEBUG [Thread-799] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:54:22,792 DEBUG [Thread-799] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:54:22,792 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:22,792 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:22,792 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:22,792 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:22,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:22,892 DEBUG [Thread-799] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:54:22,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:54:22,894 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:54:22,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:54:22,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:54:22,895 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:54:22,896 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:22,896 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:54:22,896 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:54:22,896 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:22,896 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:22,897 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:22,897 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:22,897 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:22,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:22,898 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:22,898 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:54:22,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:22,898 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:22,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:54:22,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:22,898 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:54:22,899 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:54:22,899 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:54:22,899 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:54:22,899 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:22,899 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:54:22,899 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:54:22,899 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:54:22,899 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. 2023-04-25 02:54:22,899 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. started... 2023-04-25 02:54:22,899 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. as already flushing 2023-04-25 02:54:22,899 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. 2023-04-25 02:54:22,899 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:54:22,899 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:54:22,899 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:22,899 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:54:22,899 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:54:22,902 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:22,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:22,902 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:22,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:22,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:22,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:22,902 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:22,902 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:54:22,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:22,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:22,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:22,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:22,903 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:54:22,903 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3ed1a359[Count = 0] remaining members to acquire global barrier 2023-04-25 02:54:22,903 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:54:22,903 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:22,905 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:22,905 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:22,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:22,905 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:54:22,905 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:54:22,905 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:54:22,905 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:22,905 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:54:22,906 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:54:22,906 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:22,906 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:22,906 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:54:22,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:22,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:22,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:22,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:22,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:22,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:22,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:22,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:22,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:22,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:22,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:54:22,909 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:54:22,909 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:54:22,909 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:54:22,909 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:54:22,909 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:54:22,909 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:54:22,910 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:22,910 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:22,910 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:22,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:22,910 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:22,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:22,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:22,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:22,911 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:22,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:22,911 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:22,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:22,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:22,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:22,912 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:22,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:22,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:22,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:22,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:22,915 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:22,915 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:22,915 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:22,915 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:22,915 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:22,915 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:54:22,916 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:54:22,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:22,915 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:22,916 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:54:22,916 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:22,916 DEBUG [Thread-799] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:54:22,916 DEBUG [Thread-799] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:54:22,916 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:22,916 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:22,916 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:22,916 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:22,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:22,916 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:22,958 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:22,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53664 deadline: 1682391322956, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:22,964 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:22,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53692 deadline: 1682391322961, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:23,016 DEBUG [Thread-799] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:54:23,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:54:23,018 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:54:23,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:54:23,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:54:23,020 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:54:23,020 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:23,020 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:54:23,020 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:54:23,020 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:23,020 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:23,021 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:23,021 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:23,022 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:23,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:23,022 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:23,022 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:54:23,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:23,022 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:23,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:54:23,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:23,023 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:54:23,023 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:54:23,023 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:54:23,023 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:54:23,023 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:23,023 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:54:23,023 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:54:23,023 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:54:23,023 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. 2023-04-25 02:54:23,023 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. started... 2023-04-25 02:54:23,023 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. as already flushing 2023-04-25 02:54:23,023 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. 2023-04-25 02:54:23,023 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:54:23,024 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:54:23,024 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:23,024 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:54:23,024 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:54:23,025 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:23,025 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:23,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:23,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:23,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:23,026 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:23,026 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:54:23,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:23,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:23,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:23,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:23,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:23,028 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:54:23,028 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@29263e06[Count = 0] remaining members to acquire global barrier 2023-04-25 02:54:23,028 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:54:23,028 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:23,029 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:23,029 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:23,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:23,029 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:54:23,029 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:54:23,029 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:54:23,029 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:23,029 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:54:23,032 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:54:23,032 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:23,032 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:23,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:23,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:23,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:23,032 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:54:23,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:23,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:23,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:23,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:23,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:23,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:23,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:23,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:54:23,034 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:54:23,034 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:54:23,034 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:54:23,034 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:54:23,034 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:54:23,034 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:54:23,036 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:23,036 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:23,036 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:23,036 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:23,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:23,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:23,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:23,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:23,036 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:23,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:23,036 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:23,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:23,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:23,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:23,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:23,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:23,038 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:23,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:23,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:23,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:23,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:23,040 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:23,040 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:23,040 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:23,040 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:54:23,040 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:23,040 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:23,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:23,040 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:54:23,040 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:54:23,040 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:23,040 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:23,041 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:23,041 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:23,041 DEBUG [Thread-799] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:54:23,041 DEBUG [Thread-799] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:54:23,041 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:23,041 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:23,041 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:23,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:23,141 DEBUG [Thread-799] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:54:23,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:54:23,143 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:54:23,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:54:23,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:54:23,144 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:54:23,144 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:23,144 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:54:23,144 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:54:23,144 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:23,144 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:23,145 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:23,145 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:23,146 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:23,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:23,146 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:23,146 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:54:23,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:23,146 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:23,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:54:23,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:23,147 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:54:23,147 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:54:23,147 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:54:23,147 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:54:23,147 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:23,147 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:54:23,148 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:54:23,148 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:54:23,148 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. 2023-04-25 02:54:23,148 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. started... 2023-04-25 02:54:23,148 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. as already flushing 2023-04-25 02:54:23,148 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. 2023-04-25 02:54:23,148 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:54:23,148 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:54:23,148 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:23,148 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:54:23,148 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:54:23,150 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:23,150 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:23,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:23,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:23,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:23,151 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:23,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:23,151 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:54:23,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:23,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:23,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:23,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:23,152 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:54:23,152 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4007a57d[Count = 0] remaining members to acquire global barrier 2023-04-25 02:54:23,152 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:54:23,152 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:23,153 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:23,153 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:23,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:23,153 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:54:23,153 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:23,153 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:54:23,153 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:54:23,153 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:54:23,155 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:54:23,155 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:23,155 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:23,155 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:54:23,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:23,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:23,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:23,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:23,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:23,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:23,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:23,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:23,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:23,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:23,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:54:23,157 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:54:23,157 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:54:23,157 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:54:23,157 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:54:23,157 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:54:23,157 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:54:23,162 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:23,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53610 deadline: 1682391323160, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:23,164 DEBUG [Thread-797] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4129 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198., hostname=jenkins-hbase4.apache.org,46697,1682391220023, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor28.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-25 02:54:23,165 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:23,165 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:23,165 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:23,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:23,165 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:23,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:23,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:23,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:23,165 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:23,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:23,165 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:23,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:23,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:23,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:23,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:23,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:23,167 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:23,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:23,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:23,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:23,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:23,169 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:23,169 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:23,169 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:23,169 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:54:23,169 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:54:23,169 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:54:23,169 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:23,169 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:23,169 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:23,169 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:23,169 DEBUG [Thread-799] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:54:23,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:23,169 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:23,169 DEBUG [Thread-799] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:54:23,169 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:23,169 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:23,169 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:23,170 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:23,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:23,178 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=260 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/.tmp/B/107b8acc22fb42ec84647cfd2033d076 2023-04-25 02:54:23,193 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:23,193 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53642 deadline: 1682391323192, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:23,195 DEBUG [Thread-789] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8163 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198., hostname=jenkins-hbase4.apache.org,46697,1682391220023, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor28.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-25 02:54:23,214 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:23,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53650 deadline: 1682391323214, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:23,216 DEBUG [Thread-793] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8182 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198., hostname=jenkins-hbase4.apache.org,46697,1682391220023, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor28.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-25 02:54:23,269 DEBUG [Thread-799] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:54:23,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:54:23,272 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:54:23,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:54:23,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:54:23,273 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:54:23,273 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:23,273 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:54:23,273 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:54:23,274 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:23,274 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:23,275 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:23,275 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:23,275 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:23,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:23,275 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:23,275 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:54:23,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:23,276 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:23,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:54:23,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:23,276 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:54:23,276 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:54:23,276 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:54:23,276 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:54:23,276 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:23,277 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:54:23,277 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:54:23,277 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:54:23,277 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. 2023-04-25 02:54:23,277 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. started... 2023-04-25 02:54:23,277 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. as already flushing 2023-04-25 02:54:23,277 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. 2023-04-25 02:54:23,277 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:54:23,277 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:54:23,277 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:23,277 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:54:23,277 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:54:23,278 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:23,278 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:23,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:23,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:23,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:23,279 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:23,279 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:54:23,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:23,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:23,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:23,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:23,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:23,280 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:54:23,280 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:54:23,280 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@77c50b41[Count = 0] remaining members to acquire global barrier 2023-04-25 02:54:23,280 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:23,282 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:23,282 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:23,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:23,283 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:54:23,283 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:54:23,283 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:54:23,283 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:23,283 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:54:23,284 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:54:23,284 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:23,284 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:23,284 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:54:23,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:23,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:23,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:23,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:23,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:23,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:23,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:23,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:23,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:23,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:23,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:54:23,286 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:54:23,286 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:54:23,286 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:54:23,286 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:54:23,286 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:54:23,286 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:54:23,287 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:23,287 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:23,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:23,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:23,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:23,287 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:23,287 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:23,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:23,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:23,288 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:23,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:23,288 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:23,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:23,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:23,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:23,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:23,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:23,289 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:23,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:23,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:23,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:23,291 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:23,291 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:23,291 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:23,291 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:23,291 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:23,291 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:23,291 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:23,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:23,291 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:54:23,291 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:54:23,291 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:23,291 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:23,291 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:23,291 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:23,292 DEBUG [Thread-799] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:54:23,292 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:23,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:23,292 DEBUG [Thread-799] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:54:23,392 DEBUG [Thread-799] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:54:23,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:54:23,394 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:54:23,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:54:23,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:54:23,395 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:54:23,395 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:23,395 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:54:23,395 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:54:23,396 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:23,396 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:23,398 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:23,398 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:23,398 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:23,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:23,399 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:23,399 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:54:23,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:23,399 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:23,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:54:23,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:23,399 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:54:23,400 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:54:23,400 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:54:23,400 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:54:23,400 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:23,400 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:54:23,400 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:54:23,400 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:54:23,400 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. 2023-04-25 02:54:23,400 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. started... 2023-04-25 02:54:23,400 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. as already flushing 2023-04-25 02:54:23,400 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. 2023-04-25 02:54:23,400 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:54:23,400 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:54:23,400 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:23,400 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:54:23,400 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:54:23,402 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:23,402 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:23,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:23,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:23,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:23,402 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:23,402 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:54:23,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:23,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:23,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:23,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:23,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:23,403 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:54:23,404 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@443da58b[Count = 0] remaining members to acquire global barrier 2023-04-25 02:54:23,404 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:54:23,404 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:23,405 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:23,405 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:23,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:23,405 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:54:23,405 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:54:23,405 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:54:23,405 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:23,405 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:54:23,406 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:23,406 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:54:23,406 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:23,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:23,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:23,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:23,406 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:54:23,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:23,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:23,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:23,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:23,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:23,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:23,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:23,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:54:23,408 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:54:23,408 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:54:23,409 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:54:23,409 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:54:23,409 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:54:23,409 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:54:23,413 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:23,413 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:23,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:23,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:23,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:23,413 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:23,413 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:23,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:23,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:23,414 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:23,414 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:23,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:23,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:23,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:23,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:23,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:23,415 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:23,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:23,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:23,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:23,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:23,417 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:23,417 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:23,417 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:23,417 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:54:23,417 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:54:23,417 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:23,417 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:23,417 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:23,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:23,417 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:23,417 DEBUG [Thread-799] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:54:23,417 DEBUG [Thread-799] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:54:23,417 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:54:23,417 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:23,417 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:23,418 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:23,418 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:23,417 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:23,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:23,459 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:23,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53664 deadline: 1682391323459, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:23,466 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:23,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53692 deadline: 1682391323466, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:23,517 DEBUG [Thread-799] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:54:23,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:54:23,520 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:54:23,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:54:23,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:54:23,521 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:54:23,521 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:23,521 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:54:23,521 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:54:23,521 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:23,522 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:23,523 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:23,523 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:23,523 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:23,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:23,523 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:23,523 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:54:23,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:23,524 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:23,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:54:23,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:23,524 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:54:23,524 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:54:23,524 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:54:23,524 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:54:23,524 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:23,524 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:54:23,525 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:54:23,525 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:54:23,525 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. 2023-04-25 02:54:23,525 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. started... 2023-04-25 02:54:23,525 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. as already flushing 2023-04-25 02:54:23,525 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. 2023-04-25 02:54:23,525 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:54:23,525 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:54:23,525 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:23,525 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:54:23,525 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:54:23,526 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:23,526 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:23,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:23,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:23,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:23,527 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:23,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:23,527 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:54:23,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:23,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:23,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:23,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:23,528 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:54:23,528 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5c737504[Count = 0] remaining members to acquire global barrier 2023-04-25 02:54:23,528 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:54:23,528 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:23,530 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:23,530 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:23,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:23,531 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:54:23,531 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:54:23,531 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:23,531 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:54:23,531 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:54:23,532 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:23,532 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:54:23,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:23,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:23,532 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:23,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:23,532 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:54:23,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:23,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:23,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:23,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:23,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:23,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:23,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:23,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:54:23,534 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:54:23,534 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:54:23,534 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:54:23,534 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:54:23,534 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:54:23,534 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:54:23,536 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:23,536 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:23,536 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:23,536 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:23,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:23,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:23,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:23,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:23,537 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:23,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:23,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:23,537 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:23,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:23,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:23,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:23,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:23,538 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:23,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:23,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:23,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:23,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:23,545 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:23,545 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:23,545 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:23,545 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:23,545 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:23,545 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:54:23,545 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:54:23,545 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:54:23,545 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:23,545 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:23,545 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:23,546 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:23,546 DEBUG [Thread-799] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:54:23,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:23,546 DEBUG [Thread-799] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:54:23,546 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:23,546 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:23,546 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:23,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:23,602 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=260 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/.tmp/C/20d346642b6a4da487b41cdbd8dff658 2023-04-25 02:54:23,607 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/.tmp/A/f9a9473b993647ab9c4f0bf85254b463 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/A/f9a9473b993647ab9c4f0bf85254b463 2023-04-25 02:54:23,612 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/A/f9a9473b993647ab9c4f0bf85254b463, entries=150, sequenceid=260, filesize=25.4 K 2023-04-25 02:54:23,613 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/.tmp/B/107b8acc22fb42ec84647cfd2033d076 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/B/107b8acc22fb42ec84647cfd2033d076 2023-04-25 02:54:23,617 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/B/107b8acc22fb42ec84647cfd2033d076, entries=150, sequenceid=260, filesize=11.9 K 2023-04-25 02:54:23,618 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/.tmp/C/20d346642b6a4da487b41cdbd8dff658 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/C/20d346642b6a4da487b41cdbd8dff658 2023-04-25 02:54:23,622 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/C/20d346642b6a4da487b41cdbd8dff658, entries=150, sequenceid=260, filesize=11.9 K 2023-04-25 02:54:23,622 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~379.92 KB/389040, currentSize=46.96 KB/48090 for 388bc1373cd34a10ba8be8ddc731e198 in 1290ms, sequenceid=260, compaction requested=true 2023-04-25 02:54:23,623 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 388bc1373cd34a10ba8be8ddc731e198: 2023-04-25 02:54:23,623 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 02:54:23,623 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-25 02:54:23,623 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-25 02:54:23,623 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 02:54:23,623 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 02:54:23,624 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 104026 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-25 02:54:23,624 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HStore(1912): 388bc1373cd34a10ba8be8ddc731e198/A is initiating minor compaction (all files) 2023-04-25 02:54:23,624 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 388bc1373cd34a10ba8be8ddc731e198/A in TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. 2023-04-25 02:54:23,624 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/A/2eea0cf801f9417594e3f3f0a6dd3c7e, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/A/0f7249ffa7844ca7adae49a2cd57a4f6, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/A/7081619b390a44a9b79a1aa91b66d7d6, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/A/f9a9473b993647ab9c4f0bf85254b463] into tmpdir=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/.tmp, totalSize=101.6 K 2023-04-25 02:54:23,625 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48586 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-25 02:54:23,625 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HStore(1912): 388bc1373cd34a10ba8be8ddc731e198/B is initiating minor compaction (all files) 2023-04-25 02:54:23,625 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting 2eea0cf801f9417594e3f3f0a6dd3c7e, keycount=150, bloomtype=ROW, size=25.6 K, encoding=NONE, compression=NONE, seqNum=202, earliestPutTs=1682391257936 2023-04-25 02:54:23,625 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 388bc1373cd34a10ba8be8ddc731e198/B in TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. 2023-04-25 02:54:23,625 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/B/522cb9c8f84b426181ee9bb7dc9fc27e, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/B/0ab45ceb5c454b62b269dd7d54345b64, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/B/4c8aee74a42f4267aae1cfc5c4096dc6, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/B/107b8acc22fb42ec84647cfd2033d076] into tmpdir=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/.tmp, totalSize=47.4 K 2023-04-25 02:54:23,625 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting 0f7249ffa7844ca7adae49a2cd57a4f6, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=223, earliestPutTs=1682391259018 2023-04-25 02:54:23,625 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting 522cb9c8f84b426181ee9bb7dc9fc27e, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=202, earliestPutTs=1682391257936 2023-04-25 02:54:23,626 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting 7081619b390a44a9b79a1aa91b66d7d6, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=238, earliestPutTs=1682391260074 2023-04-25 02:54:23,626 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting 0ab45ceb5c454b62b269dd7d54345b64, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=223, earliestPutTs=1682391259018 2023-04-25 02:54:23,626 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting f9a9473b993647ab9c4f0bf85254b463, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=260, earliestPutTs=1682391262221 2023-04-25 02:54:23,626 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting 4c8aee74a42f4267aae1cfc5c4096dc6, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=238, earliestPutTs=1682391260074 2023-04-25 02:54:23,627 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting 107b8acc22fb42ec84647cfd2033d076, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=260, earliestPutTs=1682391262221 2023-04-25 02:54:23,635 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] throttle.PressureAwareThroughputController(145): 388bc1373cd34a10ba8be8ddc731e198#B#compaction#142 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-25 02:54:23,639 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] throttle.PressureAwareThroughputController(145): 388bc1373cd34a10ba8be8ddc731e198#A#compaction#141 average throughput is 3.94 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-25 02:54:23,645 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/.tmp/B/55550be5974a4c6c82dc75370e638ac3 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/B/55550be5974a4c6c82dc75370e638ac3 2023-04-25 02:54:23,646 DEBUG [Thread-799] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:54:23,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:54:23,649 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:54:23,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:54:23,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:54:23,650 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:54:23,650 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:23,650 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:54:23,650 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:54:23,650 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 388bc1373cd34a10ba8be8ddc731e198/B of 388bc1373cd34a10ba8be8ddc731e198 into 55550be5974a4c6c82dc75370e638ac3(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 02:54:23,650 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 388bc1373cd34a10ba8be8ddc731e198: 2023-04-25 02:54:23,650 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198., storeName=388bc1373cd34a10ba8be8ddc731e198/B, priority=12, startTime=1682391263623; duration=0sec 2023-04-25 02:54:23,650 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 02:54:23,650 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-25 02:54:23,650 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:23,650 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:23,652 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:23,652 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:23,652 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:23,652 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48586 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-25 02:54:23,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:23,652 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HStore(1912): 388bc1373cd34a10ba8be8ddc731e198/C is initiating minor compaction (all files) 2023-04-25 02:54:23,652 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 388bc1373cd34a10ba8be8ddc731e198/C in TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. 2023-04-25 02:54:23,652 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/C/6e9b68a842b74489b4d23700a95e60dd, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/C/14590354535f4d76854ba539310fa1ec, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/C/a6645645b71c40658dd3f3c2f72ad5ff, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/C/20d346642b6a4da487b41cdbd8dff658] into tmpdir=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/.tmp, totalSize=47.4 K 2023-04-25 02:54:23,653 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting 6e9b68a842b74489b4d23700a95e60dd, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=202, earliestPutTs=1682391257936 2023-04-25 02:54:23,653 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting 14590354535f4d76854ba539310fa1ec, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=223, earliestPutTs=1682391259018 2023-04-25 02:54:23,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:23,654 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:23,654 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting a6645645b71c40658dd3f3c2f72ad5ff, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=238, earliestPutTs=1682391260074 2023-04-25 02:54:23,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:54:23,654 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting 20d346642b6a4da487b41cdbd8dff658, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=260, earliestPutTs=1682391262221 2023-04-25 02:54:23,654 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:23,654 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:54:23,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:23,654 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:54:23,655 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:54:23,655 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:54:23,655 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:54:23,655 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:23,655 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:54:23,655 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:54:23,655 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:54:23,655 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. 2023-04-25 02:54:23,655 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. started... 2023-04-25 02:54:23,656 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 388bc1373cd34a10ba8be8ddc731e198 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-25 02:54:23,656 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 388bc1373cd34a10ba8be8ddc731e198, store=A 2023-04-25 02:54:23,656 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:54:23,656 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 388bc1373cd34a10ba8be8ddc731e198, store=B 2023-04-25 02:54:23,656 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:54:23,656 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 388bc1373cd34a10ba8be8ddc731e198, store=C 2023-04-25 02:54:23,656 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:54:23,664 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] throttle.PressureAwareThroughputController(145): 388bc1373cd34a10ba8be8ddc731e198#C#compaction#143 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-25 02:54:23,683 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/.tmp/C/e753733aec9a45de8dd3e15bead75896 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/C/e753733aec9a45de8dd3e15bead75896 2023-04-25 02:54:23,688 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230425b5687f1285094b4aa00f8c27b639a8d5 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425b5687f1285094b4aa00f8c27b639a8d5 2023-04-25 02:54:23,689 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 388bc1373cd34a10ba8be8ddc731e198/C of 388bc1373cd34a10ba8be8ddc731e198 into e753733aec9a45de8dd3e15bead75896(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 02:54:23,690 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 388bc1373cd34a10ba8be8ddc731e198: 2023-04-25 02:54:23,690 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198., storeName=388bc1373cd34a10ba8be8ddc731e198/C, priority=12, startTime=1682391263623; duration=0sec 2023-04-25 02:54:23,690 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 02:54:23,696 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=271, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/.tmp/A/2c1b6a186a05455eae6e2bb8021b7ad7 2023-04-25 02:54:23,708 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=271 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/.tmp/B/5e8755f5eada409bb76a1b107a3795a3 2023-04-25 02:54:24,122 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=271 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/.tmp/C/d21a7e227b4d48eab8f1543506ee34bb 2023-04-25 02:54:24,127 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/.tmp/A/2c1b6a186a05455eae6e2bb8021b7ad7 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/A/2c1b6a186a05455eae6e2bb8021b7ad7 2023-04-25 02:54:24,132 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/A/2c1b6a186a05455eae6e2bb8021b7ad7, entries=150, sequenceid=271, filesize=25.4 K 2023-04-25 02:54:24,134 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/.tmp/B/5e8755f5eada409bb76a1b107a3795a3 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/B/5e8755f5eada409bb76a1b107a3795a3 2023-04-25 02:54:24,139 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/B/5e8755f5eada409bb76a1b107a3795a3, entries=150, sequenceid=271, filesize=11.9 K 2023-04-25 02:54:24,140 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/.tmp/C/d21a7e227b4d48eab8f1543506ee34bb as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/C/d21a7e227b4d48eab8f1543506ee34bb 2023-04-25 02:54:24,145 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/C/d21a7e227b4d48eab8f1543506ee34bb, entries=150, sequenceid=271, filesize=11.9 K 2023-04-25 02:54:24,146 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=0 B/0 for 388bc1373cd34a10ba8be8ddc731e198 in 490ms, sequenceid=271, compaction requested=false 2023-04-25 02:54:24,146 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 388bc1373cd34a10ba8be8ddc731e198: 2023-04-25 02:54:24,146 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. 2023-04-25 02:54:24,146 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:54:24,146 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:54:24,146 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:24,146 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:54:24,146 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:54:24,148 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:24,148 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:24,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:24,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:24,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:24,148 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:24,148 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:54:24,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:24,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:24,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:24,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:24,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:24,150 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:54:24,150 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@21220fba[Count = 0] remaining members to acquire global barrier 2023-04-25 02:54:24,150 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:54:24,150 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:24,151 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:24,152 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:24,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:24,152 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:54:24,152 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:54:24,152 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:54:24,152 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:24,152 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:54:24,154 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:54:24,154 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:24,154 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:24,154 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:54:24,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:24,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:24,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:24,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:24,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:24,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:24,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:24,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:24,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:24,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:24,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:54:24,158 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:54:24,158 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:54:24,158 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:54:24,158 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:54:24,158 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:54:24,158 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:54:24,160 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:24,160 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:24,160 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:24,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:24,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:24,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:24,160 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:24,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:24,160 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:24,160 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:24,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:24,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:24,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:24,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:24,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:24,162 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:24,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:24,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:24,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:24,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:24,165 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:24,165 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:24,165 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:24,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:24,165 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:24,165 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:24,165 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:54:24,165 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:54:24,165 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:24,165 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:24,165 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:54:24,165 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:24,165 DEBUG [Thread-799] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:54:24,165 DEBUG [Thread-799] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:54:24,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:24,165 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:24,165 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:24,165 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:24,166 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:24,265 DEBUG [Thread-799] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:54:24,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:54:24,268 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:54:24,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:54:24,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:54:24,269 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:54:24,270 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:24,270 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:54:24,270 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:54:24,270 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:24,270 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:24,271 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:24,271 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:24,271 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:24,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:24,272 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:24,272 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:54:24,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:24,272 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:24,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:54:24,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:24,272 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:54:24,273 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:54:24,273 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:54:24,273 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:54:24,273 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:24,273 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:54:24,273 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:54:24,273 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:54:24,273 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. 2023-04-25 02:54:24,273 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. started... 2023-04-25 02:54:24,273 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 388bc1373cd34a10ba8be8ddc731e198: 2023-04-25 02:54:24,273 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. 2023-04-25 02:54:24,274 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:54:24,274 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:54:24,274 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:24,274 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:54:24,274 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:54:24,278 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:24,278 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:24,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:24,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:24,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:24,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:24,279 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:24,279 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:54:24,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:24,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:24,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:24,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:24,281 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:54:24,281 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7ee287dc[Count = 0] remaining members to acquire global barrier 2023-04-25 02:54:24,281 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:54:24,281 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:24,282 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:24,282 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:24,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:24,282 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:54:24,282 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:54:24,282 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:24,282 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:54:24,282 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:54:24,284 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:54:24,284 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:24,284 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:54:24,284 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:24,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:24,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:24,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:24,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:24,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:24,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:24,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:24,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:24,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:24,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:24,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:54:24,287 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:54:24,287 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:54:24,287 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:54:24,287 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:54:24,287 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:54:24,287 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:54:24,289 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:24,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:24,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:24,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:24,289 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:24,289 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:24,289 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:24,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:24,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:24,289 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:24,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:24,290 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:24,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:24,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:24,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:24,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:24,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:24,291 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:24,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:24,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:24,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:24,295 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:24,295 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:24,295 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:24,295 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:24,295 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:24,295 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:24,295 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:24,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:24,295 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:54:24,296 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:54:24,295 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:24,295 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:24,296 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:24,296 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:24,296 DEBUG [Thread-799] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:54:24,296 DEBUG [Thread-799] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:54:24,296 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:24,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:24,396 DEBUG [Thread-799] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:54:24,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:54:24,398 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:54:24,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:54:24,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:54:24,400 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:54:24,400 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:24,400 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:54:24,400 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:54:24,400 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:24,401 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:24,402 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:24,402 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:24,402 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:24,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:24,402 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:24,402 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:54:24,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:24,403 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:24,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:54:24,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:24,403 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:54:24,404 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:54:24,404 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:54:24,404 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:54:24,404 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:24,404 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:54:24,404 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:54:24,404 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:54:24,404 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. 2023-04-25 02:54:24,404 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. started... 2023-04-25 02:54:24,404 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 388bc1373cd34a10ba8be8ddc731e198: 2023-04-25 02:54:24,404 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. 2023-04-25 02:54:24,404 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:54:24,404 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:54:24,404 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:24,405 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:54:24,405 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:54:24,406 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:24,406 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:24,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:24,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:24,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:24,406 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:24,406 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:54:24,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:24,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:24,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:24,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:24,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:24,408 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:54:24,408 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@575a7476[Count = 0] remaining members to acquire global barrier 2023-04-25 02:54:24,408 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:54:24,408 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:24,409 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:24,409 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:24,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:24,409 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:54:24,409 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:54:24,410 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:54:24,410 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:24,410 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:54:24,412 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:24,412 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:54:24,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:24,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:24,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:24,412 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:24,412 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:54:24,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:24,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:24,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:24,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:24,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:24,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:24,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:24,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:54:24,415 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:54:24,415 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:54:24,415 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:54:24,415 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:54:24,415 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:54:24,415 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:54:24,417 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:24,417 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:24,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:24,417 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:24,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:24,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:24,417 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:24,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:24,418 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:24,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:24,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:24,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:24,418 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:24,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:24,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:24,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:24,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:24,420 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:24,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:24,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:24,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:24,426 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:24,426 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:24,426 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:24,426 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:24,426 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:24,426 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:24,426 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:54:24,426 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:54:24,426 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:24,426 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:24,426 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:24,426 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:24,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:24,426 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:24,427 DEBUG [Thread-799] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:54:24,427 DEBUG [Thread-799] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:54:24,427 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:24,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:24,462 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/.tmp/A/044c787b709c4d3eaf1b670e394639be as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/A/044c787b709c4d3eaf1b670e394639be 2023-04-25 02:54:24,467 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 388bc1373cd34a10ba8be8ddc731e198/A of 388bc1373cd34a10ba8be8ddc731e198 into 044c787b709c4d3eaf1b670e394639be(size=25.9 K), total size for store is 51.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 02:54:24,467 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 388bc1373cd34a10ba8be8ddc731e198: 2023-04-25 02:54:24,467 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198., storeName=388bc1373cd34a10ba8be8ddc731e198/A, priority=12, startTime=1682391263623; duration=0sec 2023-04-25 02:54:24,468 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 02:54:24,527 DEBUG [Thread-799] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:54:24,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:54:24,530 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:54:24,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:54:24,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:54:24,531 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:54:24,531 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:24,531 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:54:24,531 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:54:24,532 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:24,532 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:24,534 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:24,534 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:24,534 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:24,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:24,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:24,535 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:24,535 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:54:24,535 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:24,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:54:24,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:24,536 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:54:24,536 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:54:24,536 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:54:24,536 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:54:24,536 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:24,536 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:54:24,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(9158): Flush requested on 388bc1373cd34a10ba8be8ddc731e198 2023-04-25 02:54:24,539 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 388bc1373cd34a10ba8be8ddc731e198 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-25 02:54:24,539 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 388bc1373cd34a10ba8be8ddc731e198, store=A 2023-04-25 02:54:24,540 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:54:24,540 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 388bc1373cd34a10ba8be8ddc731e198, store=B 2023-04-25 02:54:24,540 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:54:24,540 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 388bc1373cd34a10ba8be8ddc731e198, store=C 2023-04-25 02:54:24,540 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:54:24,548 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:54:24,548 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:54:24,556 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. 2023-04-25 02:54:24,557 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. started... 2023-04-25 02:54:24,557 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. as already flushing 2023-04-25 02:54:24,557 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. 2023-04-25 02:54:24,557 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:54:24,557 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:54:24,557 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:24,557 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:54:24,557 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:54:24,559 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:24,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:24,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:24,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:24,559 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:24,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:24,564 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:24,564 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:54:24,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:24,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:24,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:24,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:24,568 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:54:24,568 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@10c6f0cc[Count = 0] remaining members to acquire global barrier 2023-04-25 02:54:24,569 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:54:24,569 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:24,570 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:24,570 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:24,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:24,570 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:54:24,570 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:24,570 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:54:24,570 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:54:24,573 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:54:24,574 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:24,574 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:54:24,574 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:24,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:24,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:24,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:24,574 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:54:24,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:24,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:24,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:24,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:24,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:24,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:24,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:24,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:54:24,578 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:54:24,578 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:54:24,578 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:54:24,578 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:54:24,578 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:54:24,578 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:54:24,581 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:24,581 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:24,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:24,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:24,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:24,581 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:24,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:24,581 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:24,582 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:24,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:24,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:24,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:24,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:24,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:24,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:24,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:24,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:24,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:24,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:24,586 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:24,588 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:24,590 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:24,590 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:24,590 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:24,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:24,591 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:24,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:24,591 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:24,591 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:24,591 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:54:24,592 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:54:24,591 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:24,591 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:24,592 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:24,592 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:24,592 DEBUG [Thread-799] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:54:24,592 DEBUG [Thread-799] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:54:24,592 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:24,592 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:24,614 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=759, cellSize=34755, totalHeapSize=79176, min timestamp=1682391264557, max timestamp=1682391264612Num uniques 859; 2023-04-25 02:54:24,614 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=34755, suffix heap size=117064,new segment heap size=79176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=759 2023-04-25 02:54:24,614 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=759, cellSize=34755, totalHeapSize=79176, min timestamp=1682391264557, max timestamp=1682391264612Num uniques 859; 2023-04-25 02:54:24,614 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=34755, suffix heap size=117064,new segment heap size=79176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=759 2023-04-25 02:54:24,614 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=759, cellSize=34755, totalHeapSize=79176, min timestamp=1682391264557, max timestamp=1682391264612Num uniques 859; 2023-04-25 02:54:24,614 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=34755, suffix heap size=117064,new segment heap size=79176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=759 2023-04-25 02:54:24,638 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:24,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53692 deadline: 1682391324634, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:24,638 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:24,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53664 deadline: 1682391324634, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:24,692 DEBUG [Thread-799] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:54:24,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:54:24,694 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:54:24,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:54:24,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:54:24,696 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:54:24,696 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:24,696 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:54:24,696 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:54:24,696 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:24,696 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:24,698 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:24,698 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:24,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:24,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:24,698 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:24,698 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:54:24,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:24,698 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:24,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:54:24,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:24,699 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:54:24,699 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:54:24,699 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:54:24,699 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:24,699 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:54:24,699 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:54:24,700 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:54:24,700 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. 2023-04-25 02:54:24,700 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. started... 2023-04-25 02:54:24,700 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. as already flushing 2023-04-25 02:54:24,700 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. 2023-04-25 02:54:24,700 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:54:24,700 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:54:24,700 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:24,700 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:54:24,700 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:54:24,701 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:24,702 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:24,702 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:54:24,702 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:24,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:24,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:24,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:24,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:24,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:24,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:24,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:24,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:24,704 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:54:24,704 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@212872b1[Count = 0] remaining members to acquire global barrier 2023-04-25 02:54:24,704 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:54:24,704 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:24,706 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:24,706 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:24,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:24,706 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:54:24,706 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:54:24,706 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:54:24,707 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:24,707 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:54:24,708 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:24,708 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:54:24,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:24,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:24,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:24,708 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:24,708 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:54:24,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:24,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:24,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:24,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:24,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:24,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:24,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:24,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:54:24,711 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:54:24,711 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:54:24,711 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:54:24,711 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:54:24,711 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:54:24,711 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:54:24,713 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:24,713 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:24,713 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:24,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:24,713 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:24,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:24,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:24,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:24,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:24,713 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:24,713 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:24,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:24,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:24,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:24,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:24,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:24,716 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:24,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:24,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:24,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:24,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:24,719 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:24,719 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:24,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:24,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:24,719 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:24,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:24,719 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:54:24,719 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:54:24,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:24,719 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:24,719 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:24,719 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:24,719 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:24,720 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:24,719 DEBUG [Thread-799] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:54:24,720 DEBUG [Thread-799] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:54:24,720 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:24,720 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:24,739 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:24,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53692 deadline: 1682391324739, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:24,740 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:24,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53664 deadline: 1682391324739, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:24,820 DEBUG [Thread-799] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:54:24,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:54:24,823 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:54:24,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:54:24,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:54:24,824 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:54:24,824 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:24,824 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:54:24,824 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:54:24,825 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:24,825 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:24,827 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:24,827 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:24,827 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:24,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:24,827 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:24,827 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:54:24,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:24,828 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:24,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:54:24,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:24,828 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:54:24,828 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:54:24,828 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:54:24,828 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:54:24,829 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:24,829 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:54:24,829 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:54:24,829 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:54:24,829 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. 2023-04-25 02:54:24,829 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. started... 2023-04-25 02:54:24,829 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. as already flushing 2023-04-25 02:54:24,829 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. 2023-04-25 02:54:24,829 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:54:24,829 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:54:24,829 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:24,829 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:54:24,829 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:54:24,832 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:24,832 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:24,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:24,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:24,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:24,832 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:24,833 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:54:24,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:24,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:24,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:24,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:24,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:24,834 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:54:24,834 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:54:24,834 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4b1edd2f[Count = 0] remaining members to acquire global barrier 2023-04-25 02:54:24,834 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:24,835 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:24,836 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:24,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:24,836 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:54:24,836 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:54:24,836 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:54:24,836 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:24,836 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:54:24,837 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:24,837 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:54:24,837 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:24,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:24,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:24,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:24,837 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:54:24,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:24,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:24,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:24,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:24,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:24,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:24,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:24,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:54:24,841 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:54:24,841 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:54:24,841 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:54:24,841 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:54:24,841 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:54:24,841 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:54:24,843 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:24,843 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:24,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:24,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:24,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:24,844 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:24,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:24,844 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:24,844 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:24,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:24,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:24,844 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:24,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:24,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:24,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:24,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:24,845 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:24,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:24,847 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:24,847 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:24,848 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:24,848 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:54:24,848 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:54:24,848 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:24,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:24,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:24,848 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:24,848 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:24,848 DEBUG [Thread-799] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:54:24,848 DEBUG [Thread-799] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:54:24,848 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:24,848 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:24,848 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:24,848 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:24,848 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:24,848 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:24,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:24,848 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:54:24,944 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:24,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53692 deadline: 1682391324943, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:24,947 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:24,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53664 deadline: 1682391324946, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:24,948 DEBUG [Thread-799] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:54:24,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:54:24,951 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:54:24,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:54:24,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:54:24,952 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:54:24,952 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:24,952 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:54:24,952 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:54:24,953 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:24,953 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:24,954 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:24,954 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:24,954 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:24,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:24,954 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:24,954 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:54:24,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:24,955 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:24,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:54:24,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:24,955 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:54:24,955 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:54:24,955 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:54:24,955 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:54:24,956 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:24,956 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:54:24,956 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:54:24,956 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:54:24,956 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. 2023-04-25 02:54:24,956 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. started... 2023-04-25 02:54:24,956 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. as already flushing 2023-04-25 02:54:24,956 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. 2023-04-25 02:54:24,956 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:54:24,956 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:54:24,956 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:24,956 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:54:24,956 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:54:24,961 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:24,961 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:24,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:24,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:24,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:24,962 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:24,962 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:54:24,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:24,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:24,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:24,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:24,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:24,971 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:54:24,971 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@a5b7a05[Count = 0] remaining members to acquire global barrier 2023-04-25 02:54:24,971 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:54:24,971 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:24,972 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:24,972 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:24,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:24,972 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:24,972 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:54:24,972 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:54:24,972 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:54:24,972 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:54:24,974 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:54:24,974 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:24,974 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:24,974 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:54:24,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:24,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:24,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:24,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:24,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:24,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:24,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:24,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:24,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:24,976 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304258cab597f5b844a98acc69582b575ac88 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304258cab597f5b844a98acc69582b575ac88 2023-04-25 02:54:24,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:24,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:54:24,977 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:54:24,977 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:54:24,977 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:54:24,977 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:54:24,977 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:54:24,977 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:54:24,979 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:24,979 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:24,979 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:24,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:24,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:24,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:24,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:24,979 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:24,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:24,980 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:24,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:24,980 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:24,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:24,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:24,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:24,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:24,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:24,982 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:24,982 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=285, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/.tmp/A/3232e5011cbf44579e5238a7576bb911 2023-04-25 02:54:24,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:24,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:24,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:24,986 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:24,986 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:24,986 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:24,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:24,986 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:24,986 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:24,986 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:54:24,986 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:54:24,986 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:24,986 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:24,986 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:24,986 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:24,986 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:24,986 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:24,986 DEBUG [Thread-799] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:54:24,986 DEBUG [Thread-799] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:54:24,986 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:24,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:25,086 DEBUG [Thread-799] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:54:25,087 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:54:25,089 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:54:25,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:54:25,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:54:25,090 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:54:25,090 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:25,090 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:54:25,090 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:54:25,090 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:25,090 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:25,091 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:25,091 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:25,091 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:25,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:25,092 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:25,092 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:54:25,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:25,092 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:25,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:54:25,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:25,092 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:54:25,093 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:54:25,093 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:54:25,093 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:54:25,093 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:25,093 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:54:25,093 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:54:25,093 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:54:25,093 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. 2023-04-25 02:54:25,093 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. started... 2023-04-25 02:54:25,093 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. as already flushing 2023-04-25 02:54:25,093 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. 2023-04-25 02:54:25,093 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:54:25,093 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:54:25,093 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:25,093 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:54:25,093 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:54:25,095 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:25,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:25,095 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:25,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:25,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:25,095 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:25,095 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:54:25,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:25,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:25,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:25,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:25,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:25,096 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:54:25,096 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@371129c4[Count = 0] remaining members to acquire global barrier 2023-04-25 02:54:25,096 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:54:25,096 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:25,097 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:25,097 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:25,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:25,097 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:54:25,098 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:25,098 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:54:25,098 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:54:25,098 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:54:25,100 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:54:25,100 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:25,100 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:25,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:25,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:25,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:25,100 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:54:25,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:25,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:25,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:25,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:25,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:25,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:25,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:25,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:54:25,103 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:54:25,103 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:54:25,103 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:54:25,103 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:54:25,103 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:54:25,103 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:54:25,104 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:25,104 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:25,104 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:25,104 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:25,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:25,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:25,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:25,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:25,105 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:25,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:25,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:25,105 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:25,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:25,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:25,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:25,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:25,106 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:25,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:25,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:25,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:25,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:25,108 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:25,108 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:25,108 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:25,108 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:25,108 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:54:25,108 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:54:25,108 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:25,108 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:25,108 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:25,108 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:54:25,108 DEBUG [Thread-799] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:54:25,108 DEBUG [Thread-799] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:54:25,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:25,108 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:25,108 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:25,108 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:25,108 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:25,108 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:25,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:25,208 DEBUG [Thread-799] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:54:25,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:54:25,210 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:54:25,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:54:25,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:54:25,212 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:54:25,212 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:25,212 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:54:25,212 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:54:25,212 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:25,212 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:25,213 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:25,213 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:25,213 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:25,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:25,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:25,214 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:25,214 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:54:25,214 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:25,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:54:25,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:25,215 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:54:25,215 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:54:25,215 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:54:25,215 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:54:25,215 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:25,215 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:54:25,215 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:54:25,215 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:54:25,215 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. 2023-04-25 02:54:25,215 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. started... 2023-04-25 02:54:25,215 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. as already flushing 2023-04-25 02:54:25,215 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. 2023-04-25 02:54:25,215 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:54:25,216 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:54:25,216 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:25,216 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:54:25,216 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:54:25,218 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:25,218 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:25,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:25,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:25,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:25,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:25,218 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:25,218 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:54:25,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:25,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:25,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:25,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:25,220 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:54:25,220 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@67ed0ac1[Count = 0] remaining members to acquire global barrier 2023-04-25 02:54:25,220 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:54:25,220 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:25,221 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:25,221 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:25,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:25,221 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:54:25,221 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:25,221 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:54:25,221 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:54:25,221 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:54:25,222 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:54:25,222 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:25,222 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:25,222 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:54:25,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:25,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:25,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:25,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:25,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:25,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:25,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:25,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:25,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:25,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:25,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:54:25,224 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:54:25,224 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:54:25,224 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:54:25,224 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:54:25,224 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:54:25,224 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:54:25,226 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:25,226 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:25,226 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:25,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:25,226 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:25,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:25,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:25,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:25,226 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:25,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:25,226 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:25,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:25,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:25,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:25,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:25,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:25,227 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:25,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:25,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:25,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:25,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:25,230 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:25,230 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:25,230 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:25,230 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:25,230 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:25,230 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:54:25,231 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:54:25,230 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:25,230 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:25,231 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:25,231 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:54:25,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:25,231 DEBUG [Thread-799] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:54:25,231 DEBUG [Thread-799] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:54:25,231 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:25,231 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:25,231 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:25,231 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:25,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:25,248 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:25,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53692 deadline: 1682391325247, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:25,251 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:25,251 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53664 deadline: 1682391325250, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:25,331 DEBUG [Thread-799] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:54:25,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:54:25,333 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:54:25,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:54:25,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:54:25,334 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:54:25,334 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:25,335 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:54:25,335 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:54:25,337 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:25,337 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:25,338 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:25,338 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:25,338 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:25,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:25,338 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:25,338 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:54:25,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:25,339 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:25,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:54:25,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:25,339 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:54:25,339 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:54:25,339 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:54:25,339 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:54:25,339 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:25,339 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:54:25,340 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:54:25,340 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:54:25,340 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. 2023-04-25 02:54:25,340 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. started... 2023-04-25 02:54:25,340 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. as already flushing 2023-04-25 02:54:25,340 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. 2023-04-25 02:54:25,340 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:54:25,340 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:54:25,340 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:25,340 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:54:25,340 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:54:25,341 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:25,341 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:25,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:25,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:25,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:25,342 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:25,342 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:54:25,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:25,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:25,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:25,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:25,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:25,343 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:54:25,343 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2e225022[Count = 0] remaining members to acquire global barrier 2023-04-25 02:54:25,343 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:54:25,343 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:25,348 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:25,348 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:25,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:25,348 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:54:25,348 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:54:25,348 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:54:25,348 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:25,348 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:54:25,349 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:54:25,349 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:25,349 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:25,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:25,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:25,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:25,349 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:54:25,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:25,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:25,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:25,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:25,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:25,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:25,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:25,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:54:25,352 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:54:25,352 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:54:25,352 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:54:25,352 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:54:25,352 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:54:25,352 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:54:25,353 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:25,353 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:25,353 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:25,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:25,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:25,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:25,353 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:25,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:25,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:25,354 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:25,354 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:25,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:25,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:25,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:25,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:25,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:25,355 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:25,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:25,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:25,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:25,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:25,357 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:25,357 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:25,357 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:25,357 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:25,357 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:25,357 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:25,357 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:25,357 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:25,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:25,357 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:54:25,357 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:54:25,357 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:25,358 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:25,358 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:25,358 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:25,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:25,358 DEBUG [Thread-799] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:54:25,358 DEBUG [Thread-799] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:54:25,393 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=285 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/.tmp/B/0f07c3654503421995bc37dea0c599c9 2023-04-25 02:54:25,458 DEBUG [Thread-799] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:54:25,459 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:54:25,460 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:54:25,461 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:54:25,461 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:54:25,461 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:54:25,461 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:25,461 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:54:25,461 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:54:25,462 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:25,462 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:25,464 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:25,464 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:25,464 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:25,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:25,464 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:25,464 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:54:25,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:25,465 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:25,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:54:25,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:25,465 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:54:25,465 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:54:25,465 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:54:25,465 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:54:25,465 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:25,465 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:54:25,466 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:54:25,466 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:54:25,466 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. 2023-04-25 02:54:25,466 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. started... 2023-04-25 02:54:25,466 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. as already flushing 2023-04-25 02:54:25,466 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. 2023-04-25 02:54:25,466 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:54:25,466 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:54:25,466 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:25,466 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:54:25,466 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:54:25,467 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:25,468 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:25,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:25,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:25,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:25,468 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:25,468 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:54:25,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:25,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:25,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:25,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:25,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:25,469 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:54:25,469 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:54:25,469 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7f789cb1[Count = 0] remaining members to acquire global barrier 2023-04-25 02:54:25,469 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:25,470 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:25,470 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:25,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:25,471 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:54:25,471 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:54:25,471 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:54:25,471 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:25,471 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:54:25,472 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:25,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:25,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:25,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:25,472 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:54:25,472 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:25,472 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:54:25,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:25,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:25,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:25,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:25,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:25,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:25,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:25,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:54:25,474 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:54:25,474 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:54:25,474 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:54:25,474 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:54:25,474 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:54:25,474 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:54:25,482 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:25,482 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:25,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:25,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:25,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:25,482 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:25,482 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:25,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:25,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:25,482 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:25,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:25,482 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:25,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:25,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:25,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:25,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:25,483 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:25,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:25,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:25,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:25,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:25,486 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:25,486 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:25,486 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:25,486 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:25,486 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:25,486 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:54:25,486 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:54:25,486 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:25,486 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:25,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:25,486 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:54:25,486 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:25,486 DEBUG [Thread-799] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:54:25,486 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:25,486 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:25,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:25,486 DEBUG [Thread-799] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:54:25,486 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:25,486 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:25,587 DEBUG [Thread-799] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:54:25,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:54:25,589 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:54:25,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:54:25,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:54:25,590 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:54:25,590 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:25,590 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:54:25,590 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:54:25,590 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:25,590 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:25,591 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:25,591 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:25,591 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:25,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:25,591 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:25,591 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:54:25,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:25,592 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:25,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:54:25,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:25,592 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:54:25,592 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:54:25,592 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:54:25,593 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:54:25,593 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:25,593 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:54:25,593 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:54:25,593 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:54:25,593 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. 2023-04-25 02:54:25,593 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. started... 2023-04-25 02:54:25,593 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. as already flushing 2023-04-25 02:54:25,593 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. 2023-04-25 02:54:25,593 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:54:25,593 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:54:25,593 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:25,593 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:54:25,593 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:54:25,594 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:25,594 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:25,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:25,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:25,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:25,595 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:25,595 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:54:25,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:25,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:25,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:25,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:25,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:25,596 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:54:25,596 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:54:25,596 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4119c593[Count = 0] remaining members to acquire global barrier 2023-04-25 02:54:25,596 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:25,598 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:25,598 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:25,598 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:54:25,598 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:25,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:25,598 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:54:25,598 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:54:25,598 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:54:25,600 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:54:25,600 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:25,600 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:25,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:25,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:25,600 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:54:25,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:25,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:25,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:25,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:25,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:25,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:25,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:25,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:25,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:54:25,602 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:54:25,602 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:54:25,602 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:54:25,602 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:54:25,602 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:54:25,602 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:54:25,603 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:25,603 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:25,603 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:25,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:25,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:25,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:25,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:25,603 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:25,604 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:25,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:25,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:25,604 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:25,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:25,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:25,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:25,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:25,605 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:25,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:25,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:25,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:25,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:25,607 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:25,607 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:25,607 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:25,607 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:54:25,607 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:54:25,607 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:25,607 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:54:25,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:25,607 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:25,607 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:25,607 DEBUG [Thread-799] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:54:25,607 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:25,607 DEBUG [Thread-799] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:54:25,607 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:25,607 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:25,607 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:25,607 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:25,608 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:25,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:25,707 DEBUG [Thread-799] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:54:25,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:54:25,710 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:54:25,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:54:25,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:54:25,711 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:54:25,711 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:25,711 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:54:25,711 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:54:25,711 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:25,711 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:25,714 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:25,714 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:25,714 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:25,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:25,714 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:25,714 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:54:25,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:25,714 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:25,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:54:25,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:25,715 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:54:25,715 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:54:25,715 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:54:25,715 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:54:25,715 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:25,715 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:54:25,715 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:54:25,715 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:54:25,715 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. 2023-04-25 02:54:25,715 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. started... 2023-04-25 02:54:25,715 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. as already flushing 2023-04-25 02:54:25,716 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. 2023-04-25 02:54:25,716 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:54:25,716 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:54:25,716 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:25,716 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:54:25,716 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:54:25,717 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:25,717 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:25,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:25,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:25,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:25,717 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:25,717 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:54:25,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:25,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:25,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:25,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:25,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:25,718 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:54:25,719 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4ead7473[Count = 0] remaining members to acquire global barrier 2023-04-25 02:54:25,719 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:54:25,719 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:25,720 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:25,720 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:25,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:25,720 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:54:25,720 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:54:25,720 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:54:25,720 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:25,720 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:54:25,721 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:25,721 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:54:25,721 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:25,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:25,721 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:54:25,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:25,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:25,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:25,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:25,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:25,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:25,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:25,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:25,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:25,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:54:25,723 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:54:25,723 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:54:25,723 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:54:25,723 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:54:25,723 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:54:25,723 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:54:25,728 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:25,728 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:25,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:25,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:25,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:25,728 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:25,728 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:25,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:25,728 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:25,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:25,729 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:25,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:25,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:25,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:25,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:25,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:25,730 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:25,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:25,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:25,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:25,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:25,732 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:25,732 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:25,732 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:25,732 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:25,732 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:25,732 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:54:25,732 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:54:25,732 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:25,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:25,732 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:25,732 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:25,732 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:25,732 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:25,732 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:25,732 DEBUG [Thread-799] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:54:25,733 DEBUG [Thread-799] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:54:25,733 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:25,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:25,754 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:25,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53692 deadline: 1682391325753, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:25,755 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:25,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53664 deadline: 1682391325755, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:25,804 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=285 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/.tmp/C/0792d4b169b54c9a8b0bfed88ed368a4 2023-04-25 02:54:25,809 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/.tmp/A/3232e5011cbf44579e5238a7576bb911 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/A/3232e5011cbf44579e5238a7576bb911 2023-04-25 02:54:25,814 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/A/3232e5011cbf44579e5238a7576bb911, entries=350, sequenceid=285, filesize=52.9 K 2023-04-25 02:54:25,815 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/.tmp/B/0f07c3654503421995bc37dea0c599c9 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/B/0f07c3654503421995bc37dea0c599c9 2023-04-25 02:54:25,819 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/B/0f07c3654503421995bc37dea0c599c9, entries=100, sequenceid=285, filesize=9.5 K 2023-04-25 02:54:25,820 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/.tmp/C/0792d4b169b54c9a8b0bfed88ed368a4 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/C/0792d4b169b54c9a8b0bfed88ed368a4 2023-04-25 02:54:25,824 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/C/0792d4b169b54c9a8b0bfed88ed368a4, entries=100, sequenceid=285, filesize=9.5 K 2023-04-25 02:54:25,825 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=161.02 KB/164880 for 388bc1373cd34a10ba8be8ddc731e198 in 1286ms, sequenceid=285, compaction requested=true 2023-04-25 02:54:25,825 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 388bc1373cd34a10ba8be8ddc731e198: 2023-04-25 02:54:25,825 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 02:54:25,825 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-25 02:54:25,825 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-25 02:54:25,825 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 02:54:25,825 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 02:54:25,826 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 106730 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-25 02:54:25,826 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34542 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-25 02:54:25,827 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HStore(1912): 388bc1373cd34a10ba8be8ddc731e198/A is initiating minor compaction (all files) 2023-04-25 02:54:25,827 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HStore(1912): 388bc1373cd34a10ba8be8ddc731e198/B is initiating minor compaction (all files) 2023-04-25 02:54:25,827 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 388bc1373cd34a10ba8be8ddc731e198/A in TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. 2023-04-25 02:54:25,827 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 388bc1373cd34a10ba8be8ddc731e198/B in TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. 2023-04-25 02:54:25,827 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/A/044c787b709c4d3eaf1b670e394639be, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/A/2c1b6a186a05455eae6e2bb8021b7ad7, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/A/3232e5011cbf44579e5238a7576bb911] into tmpdir=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/.tmp, totalSize=104.2 K 2023-04-25 02:54:25,827 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/B/55550be5974a4c6c82dc75370e638ac3, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/B/5e8755f5eada409bb76a1b107a3795a3, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/B/0f07c3654503421995bc37dea0c599c9] into tmpdir=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/.tmp, totalSize=33.7 K 2023-04-25 02:54:25,827 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting 044c787b709c4d3eaf1b670e394639be, keycount=150, bloomtype=ROW, size=25.9 K, encoding=NONE, compression=NONE, seqNum=260, earliestPutTs=1682391262221 2023-04-25 02:54:25,827 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting 55550be5974a4c6c82dc75370e638ac3, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=260, earliestPutTs=1682391262221 2023-04-25 02:54:25,828 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting 2c1b6a186a05455eae6e2bb8021b7ad7, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=271, earliestPutTs=1682391262336 2023-04-25 02:54:25,828 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting 5e8755f5eada409bb76a1b107a3795a3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=271, earliestPutTs=1682391262336 2023-04-25 02:54:25,829 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting 0f07c3654503421995bc37dea0c599c9, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=285, earliestPutTs=1682391264529 2023-04-25 02:54:25,829 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting 3232e5011cbf44579e5238a7576bb911, keycount=350, bloomtype=ROW, size=52.9 K, encoding=NONE, compression=NONE, seqNum=285, earliestPutTs=1682391264492 2023-04-25 02:54:25,833 DEBUG [Thread-799] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:54:25,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:54:25,835 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:54:25,841 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:54:25,841 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:54:25,841 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:54:25,841 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:25,841 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:54:25,841 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:54:25,841 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:25,841 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:25,842 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] throttle.PressureAwareThroughputController(145): 388bc1373cd34a10ba8be8ddc731e198#B#compaction#151 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-25 02:54:25,843 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:25,843 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:25,843 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:25,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:25,844 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:25,844 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:54:25,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:25,844 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:25,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:54:25,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:25,845 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:54:25,849 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] throttle.PressureAwareThroughputController(145): 388bc1373cd34a10ba8be8ddc731e198#A#compaction#150 average throughput is 2.46 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-25 02:54:25,856 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:54:25,856 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:54:25,856 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:54:25,856 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:25,856 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:54:25,858 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:54:25,858 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:54:25,858 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. 2023-04-25 02:54:25,858 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. started... 2023-04-25 02:54:25,858 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 388bc1373cd34a10ba8be8ddc731e198 3/3 column families, dataSize=161.02 KB heapSize=408.42 KB 2023-04-25 02:54:25,858 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 388bc1373cd34a10ba8be8ddc731e198, store=A 2023-04-25 02:54:25,858 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 02:54:25,858 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 388bc1373cd34a10ba8be8ddc731e198, store=B 2023-04-25 02:54:25,859 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 02:54:25,859 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 388bc1373cd34a10ba8be8ddc731e198, store=C 2023-04-25 02:54:25,859 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 02:54:26,265 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/.tmp/B/4438dd49f6794adf822d1deab57fa16e as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/B/4438dd49f6794adf822d1deab57fa16e 2023-04-25 02:54:26,270 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 388bc1373cd34a10ba8be8ddc731e198/B of 388bc1373cd34a10ba8be8ddc731e198 into 4438dd49f6794adf822d1deab57fa16e(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 02:54:26,270 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 388bc1373cd34a10ba8be8ddc731e198: 2023-04-25 02:54:26,270 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198., storeName=388bc1373cd34a10ba8be8ddc731e198/B, priority=13, startTime=1682391265825; duration=0sec 2023-04-25 02:54:26,270 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 02:54:26,270 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-25 02:54:26,271 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34542 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-25 02:54:26,272 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HStore(1912): 388bc1373cd34a10ba8be8ddc731e198/C is initiating minor compaction (all files) 2023-04-25 02:54:26,272 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 388bc1373cd34a10ba8be8ddc731e198/C in TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. 2023-04-25 02:54:26,272 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/C/e753733aec9a45de8dd3e15bead75896, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/C/d21a7e227b4d48eab8f1543506ee34bb, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/C/0792d4b169b54c9a8b0bfed88ed368a4] into tmpdir=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/.tmp, totalSize=33.7 K 2023-04-25 02:54:26,272 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting e753733aec9a45de8dd3e15bead75896, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=260, earliestPutTs=1682391262221 2023-04-25 02:54:26,273 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting d21a7e227b4d48eab8f1543506ee34bb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=271, earliestPutTs=1682391262336 2023-04-25 02:54:26,273 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting 0792d4b169b54c9a8b0bfed88ed368a4, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=285, earliestPutTs=1682391264529 2023-04-25 02:54:26,276 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230425f5162868956248fc86720a6a083b4b04 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425f5162868956248fc86720a6a083b4b04 2023-04-25 02:54:26,285 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] throttle.PressureAwareThroughputController(145): 388bc1373cd34a10ba8be8ddc731e198#C#compaction#153 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-25 02:54:26,293 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/.tmp/C/66fc279401bf46148fb40cb78196fca4 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/C/66fc279401bf46148fb40cb78196fca4 2023-04-25 02:54:26,298 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 388bc1373cd34a10ba8be8ddc731e198/C of 388bc1373cd34a10ba8be8ddc731e198 into 66fc279401bf46148fb40cb78196fca4(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 02:54:26,298 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 388bc1373cd34a10ba8be8ddc731e198: 2023-04-25 02:54:26,298 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198., storeName=388bc1373cd34a10ba8be8ddc731e198/C, priority=13, startTime=1682391265825; duration=0sec 2023-04-25 02:54:26,298 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 02:54:26,685 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=314, memsize=53.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/.tmp/A/968a15dce17541b0877f73480fa084ba 2023-04-25 02:54:26,698 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=53.67 KB at sequenceid=314 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/.tmp/B/a72ad622ac444b9aa13ed16b5c6049fb 2023-04-25 02:54:26,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(9158): Flush requested on 388bc1373cd34a10ba8be8ddc731e198 2023-04-25 02:54:26,757 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. as already flushing 2023-04-25 02:54:26,766 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:26,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53664 deadline: 1682391326764, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:26,766 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:26,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53692 deadline: 1682391326764, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:26,869 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:26,869 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:26,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53692 deadline: 1682391326867, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:26,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53664 deadline: 1682391326867, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:27,072 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:27,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53692 deadline: 1682391327070, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:27,072 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:27,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53664 deadline: 1682391327071, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:27,081 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/.tmp/A/68b032bf259a4543b75456f2b17ac25a as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/A/68b032bf259a4543b75456f2b17ac25a 2023-04-25 02:54:27,087 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 388bc1373cd34a10ba8be8ddc731e198/A of 388bc1373cd34a10ba8be8ddc731e198 into 68b032bf259a4543b75456f2b17ac25a(size=26.0 K), total size for store is 26.0 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-04-25 02:54:27,087 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 388bc1373cd34a10ba8be8ddc731e198: 2023-04-25 02:54:27,087 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198., storeName=388bc1373cd34a10ba8be8ddc731e198/A, priority=13, startTime=1682391265825; duration=1sec 2023-04-25 02:54:27,087 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 02:54:27,110 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=53.67 KB at sequenceid=314 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/.tmp/C/6a2fcabbc73a4653822826c08ef3d003 2023-04-25 02:54:27,114 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/.tmp/A/968a15dce17541b0877f73480fa084ba as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/A/968a15dce17541b0877f73480fa084ba 2023-04-25 02:54:27,119 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/A/968a15dce17541b0877f73480fa084ba, entries=150, sequenceid=314, filesize=25.4 K 2023-04-25 02:54:27,119 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/.tmp/B/a72ad622ac444b9aa13ed16b5c6049fb as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/B/a72ad622ac444b9aa13ed16b5c6049fb 2023-04-25 02:54:27,124 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/B/a72ad622ac444b9aa13ed16b5c6049fb, entries=150, sequenceid=314, filesize=11.9 K 2023-04-25 02:54:27,125 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/.tmp/C/6a2fcabbc73a4653822826c08ef3d003 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/C/6a2fcabbc73a4653822826c08ef3d003 2023-04-25 02:54:27,130 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/C/6a2fcabbc73a4653822826c08ef3d003, entries=150, sequenceid=314, filesize=11.9 K 2023-04-25 02:54:27,131 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~161.02 KB/164880, heapSize ~408.38 KB/418176, currentSize=40.25 KB/41220 for 388bc1373cd34a10ba8be8ddc731e198 in 1273ms, sequenceid=314, compaction requested=false 2023-04-25 02:54:27,131 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 388bc1373cd34a10ba8be8ddc731e198: 2023-04-25 02:54:27,131 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. 2023-04-25 02:54:27,131 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:54:27,131 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:54:27,131 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:27,131 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:54:27,131 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:54:27,133 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:27,133 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:27,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:27,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:27,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:27,133 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:27,133 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:54:27,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:27,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:27,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:27,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:27,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:27,135 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:54:27,135 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@72b29ec4[Count = 0] remaining members to acquire global barrier 2023-04-25 02:54:27,135 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:54:27,135 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:27,136 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:27,136 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:27,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:27,137 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:54:27,137 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:54:27,137 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:54:27,137 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:27,137 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:54:27,138 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:54:27,138 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:27,138 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:27,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:27,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:27,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:27,138 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:54:27,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:27,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:27,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:27,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:27,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:27,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:27,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:27,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:54:27,140 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:54:27,140 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:54:27,141 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:54:27,141 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:54:27,141 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:54:27,141 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:54:27,142 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:27,142 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:27,142 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:27,142 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:27,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:27,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:27,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:27,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:27,142 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:27,142 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:27,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:27,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:27,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:27,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:27,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:27,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:27,144 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:27,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:27,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:27,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:27,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:27,146 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:27,146 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:27,146 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:27,146 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:27,146 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:27,146 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:27,146 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:54:27,146 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:54:27,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:27,146 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:27,146 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:27,146 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:27,146 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:27,146 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:27,146 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:54:27,146 DEBUG [Thread-799] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:54:27,146 DEBUG [Thread-799] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:54:27,146 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:27,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:27,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(9158): Flush requested on 388bc1373cd34a10ba8be8ddc731e198 2023-04-25 02:54:27,184 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 388bc1373cd34a10ba8be8ddc731e198 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-25 02:54:27,185 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 388bc1373cd34a10ba8be8ddc731e198, store=A 2023-04-25 02:54:27,185 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:54:27,185 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 388bc1373cd34a10ba8be8ddc731e198, store=B 2023-04-25 02:54:27,185 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:54:27,185 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 388bc1373cd34a10ba8be8ddc731e198, store=C 2023-04-25 02:54:27,185 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:54:27,242 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682391267212, max timestamp=1682391267240Num uniques 859; 2023-04-25 02:54:27,242 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682391267212, max timestamp=1682391267240Num uniques 859; 2023-04-25 02:54:27,242 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-25 02:54:27,242 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682391267212, max timestamp=1682391267240Num uniques 859; 2023-04-25 02:54:27,242 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-25 02:54:27,243 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-25 02:54:27,247 DEBUG [Thread-799] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:54:27,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:54:27,249 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:54:27,250 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:54:27,250 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:54:27,251 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:54:27,251 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:27,251 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:54:27,251 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:54:27,251 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:27,251 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:27,253 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:27,253 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:27,253 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:27,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:27,253 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:27,254 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:54:27,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:27,254 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:27,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:54:27,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:27,254 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:54:27,255 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:54:27,255 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:54:27,255 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:54:27,255 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:27,255 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:54:27,255 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:54:27,255 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:54:27,255 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. 2023-04-25 02:54:27,256 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. started... 2023-04-25 02:54:27,256 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. as already flushing 2023-04-25 02:54:27,256 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. 2023-04-25 02:54:27,256 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:54:27,256 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:54:27,256 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:27,256 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:54:27,256 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:54:27,257 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:27,257 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:27,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:27,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:27,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:27,257 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:27,257 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:54:27,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:27,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:27,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:27,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:27,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:27,259 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:54:27,259 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@49940937[Count = 0] remaining members to acquire global barrier 2023-04-25 02:54:27,259 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:54:27,259 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:27,260 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:27,260 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:27,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:27,260 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:54:27,260 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:54:27,260 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:54:27,260 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:27,260 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:54:27,261 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:27,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:27,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:27,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:27,261 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:54:27,261 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:27,261 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:54:27,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:27,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:27,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:27,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:27,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:27,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:27,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:27,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:54:27,264 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:54:27,264 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:54:27,264 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:54:27,264 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:54:27,264 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:54:27,264 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:54:27,267 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:27,267 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:27,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:27,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:27,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:27,267 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:27,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:27,267 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:27,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:27,267 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:27,267 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:27,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:27,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:27,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:27,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:27,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:27,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:27,268 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:27,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:27,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:27,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:27,270 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:27,271 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:27,271 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:27,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:27,271 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:27,271 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:54:27,271 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:54:27,271 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:27,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:27,271 DEBUG [Thread-799] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:54:27,271 DEBUG [Thread-799] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:54:27,271 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:27,271 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:27,271 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:27,271 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:27,271 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:27,271 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:27,271 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:27,297 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:27,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53610 deadline: 1682391327295, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:27,371 DEBUG [Thread-799] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:54:27,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:54:27,373 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:54:27,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:54:27,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:54:27,374 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:54:27,374 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:27,374 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:54:27,374 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:54:27,375 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:27,375 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:27,375 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:27,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53692 deadline: 1682391327373, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:27,376 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:27,376 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:27,376 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:27,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:27,376 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:27,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:27,376 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:54:27,377 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:27,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:54:27,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:27,377 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:54:27,377 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:54:27,377 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:54:27,377 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:54:27,377 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:27,377 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:54:27,377 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:54:27,377 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:54:27,377 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. 2023-04-25 02:54:27,377 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:27,377 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. started... 2023-04-25 02:54:27,378 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53664 deadline: 1682391327375, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:27,378 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. as already flushing 2023-04-25 02:54:27,378 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. 2023-04-25 02:54:27,378 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:54:27,378 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:54:27,378 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:27,378 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:54:27,378 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:54:27,379 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:27,379 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:27,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:27,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:27,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:27,379 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:27,379 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:54:27,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:27,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:27,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:27,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:27,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:27,381 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:54:27,381 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:54:27,381 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:27,381 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@462e68cb[Count = 0] remaining members to acquire global barrier 2023-04-25 02:54:27,383 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:27,383 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:27,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:27,383 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:54:27,383 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:54:27,383 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:54:27,383 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:27,384 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:54:27,385 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:54:27,385 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:27,385 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:27,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:27,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:27,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:27,385 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:54:27,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:27,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:27,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:27,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:27,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:27,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:27,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:27,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:54:27,387 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:54:27,387 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:54:27,387 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:54:27,387 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:54:27,387 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:54:27,387 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:54:27,388 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:27,389 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:27,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:27,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:27,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:27,389 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:27,389 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:27,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:27,389 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:27,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:27,389 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:27,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:27,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:27,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:27,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:27,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:27,390 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:27,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:27,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:27,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:27,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:27,396 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:27,396 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:27,396 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:27,396 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:27,396 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:27,397 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:27,397 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:27,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:27,397 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:54:27,396 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:27,397 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:54:27,397 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:54:27,397 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:27,397 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:27,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:27,397 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:27,397 DEBUG [Thread-799] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:54:27,397 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:27,397 DEBUG [Thread-799] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:54:27,400 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:27,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53610 deadline: 1682391327398, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:27,497 DEBUG [Thread-799] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:54:27,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:54:27,499 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:54:27,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:54:27,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:54:27,501 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:54:27,501 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:27,501 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:54:27,501 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:54:27,502 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:27,502 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:27,503 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:27,503 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:27,504 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:27,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:27,504 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:27,504 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:54:27,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:27,504 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:27,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:54:27,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:27,505 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:54:27,505 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:54:27,505 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:54:27,505 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:54:27,505 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:27,505 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:54:27,506 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:54:27,506 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:54:27,506 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. 2023-04-25 02:54:27,506 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. started... 2023-04-25 02:54:27,506 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. as already flushing 2023-04-25 02:54:27,506 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. 2023-04-25 02:54:27,506 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:54:27,506 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:54:27,506 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:27,506 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:54:27,506 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:54:27,508 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:27,508 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:27,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:27,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:27,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:27,508 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:27,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:27,508 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:54:27,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:27,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:27,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:27,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:27,509 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:54:27,509 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@63059291[Count = 0] remaining members to acquire global barrier 2023-04-25 02:54:27,509 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:54:27,510 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:27,510 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:27,510 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:27,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:27,510 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:54:27,511 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:54:27,511 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:54:27,511 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:27,511 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:54:27,513 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:27,513 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:54:27,513 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:27,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:27,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:27,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:27,513 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:54:27,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:27,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:27,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:27,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:27,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:27,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:27,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:27,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:54:27,516 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:54:27,516 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:54:27,516 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:54:27,516 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:54:27,516 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:54:27,516 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:54:27,517 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:27,517 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:27,517 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:27,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:27,517 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:27,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:27,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:27,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:27,518 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:27,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:27,518 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:27,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:27,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:27,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:27,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:27,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:27,519 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:27,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:27,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:27,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:27,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:27,521 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:27,521 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:27,521 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:27,521 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:27,521 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:27,521 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:54:27,521 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:54:27,521 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:27,521 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:27,521 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:27,521 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:27,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:27,522 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:27,522 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:27,522 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:54:27,522 DEBUG [Thread-799] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:54:27,522 DEBUG [Thread-799] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:54:27,522 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:27,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:27,598 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230425a7597d1dff174e40acab2a42c6710902 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425a7597d1dff174e40acab2a42c6710902 2023-04-25 02:54:27,603 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:27,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53610 deadline: 1682391327601, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:27,605 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=327, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/.tmp/A/084af77502304152bbfef9fec78bdc0d 2023-04-25 02:54:27,616 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=327 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/.tmp/B/fdb844eb97b3476999382e9f5e79839c 2023-04-25 02:54:27,622 DEBUG [Thread-799] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:54:27,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:54:27,624 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:54:27,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:54:27,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:54:27,625 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:54:27,626 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:27,626 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:54:27,626 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:54:27,626 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:27,626 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:27,627 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:27,627 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:27,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:27,627 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:27,628 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:27,628 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:54:27,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:27,628 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:27,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:54:27,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:27,629 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:54:27,629 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:54:27,629 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:54:27,629 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:54:27,629 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:27,629 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:54:27,629 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:54:27,629 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:54:27,629 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. 2023-04-25 02:54:27,629 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. started... 2023-04-25 02:54:27,629 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. as already flushing 2023-04-25 02:54:27,630 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. 2023-04-25 02:54:27,630 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:54:27,630 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:54:27,630 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:27,630 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:54:27,630 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:54:27,632 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:27,632 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:27,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:27,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:27,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:27,632 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:27,632 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:54:27,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:27,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:27,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:27,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:27,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:27,633 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:54:27,633 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@527748b[Count = 0] remaining members to acquire global barrier 2023-04-25 02:54:27,633 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:54:27,634 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:27,634 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:27,634 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:27,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:27,635 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:54:27,635 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:54:27,635 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:27,635 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:54:27,635 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:54:27,636 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:27,636 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:54:27,636 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:27,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:27,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:27,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:27,636 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:54:27,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:27,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:27,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:27,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:27,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:27,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:27,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:27,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:54:27,639 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:54:27,639 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:54:27,639 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:54:27,639 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:54:27,639 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:54:27,639 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:54:27,640 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:27,640 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:27,640 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:27,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:27,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:27,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:27,640 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:27,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:27,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:27,641 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:27,641 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:27,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:27,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:27,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:27,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:27,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:27,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:27,642 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:27,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:27,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:27,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:27,645 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:27,645 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:27,645 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:27,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:27,645 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:27,645 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:54:27,645 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:54:27,645 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:27,645 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:27,645 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:27,645 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:27,645 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:27,645 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:27,646 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:27,650 DEBUG [Thread-799] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:54:27,650 DEBUG [Thread-799] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:54:27,650 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:27,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:27,750 DEBUG [Thread-799] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:54:27,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:54:27,753 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:54:27,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:54:27,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:54:27,754 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:54:27,754 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:27,754 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:54:27,754 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:54:27,754 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:27,754 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:27,755 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:27,755 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:27,755 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:27,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:27,756 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:27,756 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:54:27,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:27,756 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:27,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:54:27,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:27,757 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:54:27,757 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:54:27,757 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:54:27,757 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:54:27,757 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:27,757 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:54:27,757 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:54:27,757 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:54:27,757 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. 2023-04-25 02:54:27,757 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. started... 2023-04-25 02:54:27,757 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. as already flushing 2023-04-25 02:54:27,757 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. 2023-04-25 02:54:27,757 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:54:27,757 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:54:27,758 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:27,758 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:54:27,758 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:54:27,759 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:27,759 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:27,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:27,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:27,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:27,759 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:27,759 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:54:27,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:27,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:27,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:27,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:27,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:27,761 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:54:27,761 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6c1308a9[Count = 0] remaining members to acquire global barrier 2023-04-25 02:54:27,761 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:54:27,761 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:27,762 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:27,762 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:27,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:27,762 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:54:27,762 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:54:27,762 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:54:27,762 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:27,762 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:54:27,764 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:54:27,764 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:27,764 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:27,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:27,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:27,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:27,764 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:54:27,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:27,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:27,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:27,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:27,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:27,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:27,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:27,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:54:27,766 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:54:27,766 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:54:27,767 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:54:27,767 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:54:27,767 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:54:27,767 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:54:27,768 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:27,768 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:27,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:27,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:27,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:27,768 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:27,768 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:27,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:27,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:27,768 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:27,769 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:27,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:27,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:27,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:27,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:27,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:27,770 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:27,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:27,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:27,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:27,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:27,774 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:27,774 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:27,774 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:27,774 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:27,774 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:27,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:27,774 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:54:27,775 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:54:27,774 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:27,774 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:27,775 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:27,775 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:27,775 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:27,775 DEBUG [Thread-799] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:54:27,775 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:27,775 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:27,775 DEBUG [Thread-799] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:54:27,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:27,775 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:54:27,875 DEBUG [Thread-799] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:54:27,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:54:27,878 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:54:27,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:54:27,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:54:27,879 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:54:27,879 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:27,879 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:54:27,879 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:54:27,879 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:27,879 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:27,879 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:27,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53664 deadline: 1682391327879, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:27,881 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:27,882 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:27,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:27,882 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:27,882 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:27,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53692 deadline: 1682391327879, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:27,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:27,882 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:27,882 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:54:27,882 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:27,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:54:27,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:27,883 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:54:27,883 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:54:27,883 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:54:27,883 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:54:27,883 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:27,883 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:54:27,883 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:54:27,883 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:54:27,884 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. 2023-04-25 02:54:27,884 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. started... 2023-04-25 02:54:27,884 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. as already flushing 2023-04-25 02:54:27,884 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. 2023-04-25 02:54:27,884 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:54:27,884 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:54:27,884 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:27,884 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:54:27,884 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:54:27,885 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:27,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:27,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:27,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:27,885 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:27,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:27,886 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:27,886 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:54:27,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:27,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:27,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:27,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:27,887 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:54:27,887 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@59ce5dd0[Count = 0] remaining members to acquire global barrier 2023-04-25 02:54:27,887 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:54:27,887 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:27,888 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:27,888 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:27,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:27,888 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:27,888 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:54:27,888 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:54:27,888 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:54:27,888 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:54:27,890 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:27,890 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:54:27,890 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:27,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:27,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:27,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:27,890 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:54:27,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:27,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:27,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:27,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:27,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:27,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:27,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:27,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:54:27,892 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:54:27,892 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:54:27,892 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:54:27,892 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:54:27,892 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:54:27,892 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:54:27,894 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:27,894 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:27,894 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:27,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:27,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:27,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:27,894 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:27,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:27,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:27,895 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:27,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:27,895 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:27,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:27,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:27,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:27,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:27,896 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:27,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:27,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:27,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:27,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:27,898 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:27,898 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:27,898 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:27,898 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:27,898 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:54:27,898 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:54:27,898 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:27,898 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:27,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:27,898 DEBUG [Thread-799] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:54:27,898 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:27,898 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:27,898 DEBUG [Thread-799] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:54:27,898 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:27,898 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:27,898 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:27,899 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:27,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:27,906 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:27,906 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53610 deadline: 1682391327905, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:27,998 DEBUG [Thread-799] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:54:27,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:54:28,000 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:54:28,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:54:28,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:54:28,001 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:54:28,002 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:28,002 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:54:28,002 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:54:28,002 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:28,002 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:28,003 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:28,003 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:28,003 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:28,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:28,003 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:28,003 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:54:28,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:28,004 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:28,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:54:28,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:28,004 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:54:28,004 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:54:28,004 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:54:28,005 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:54:28,005 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:28,005 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:54:28,005 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:54:28,005 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:54:28,005 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. 2023-04-25 02:54:28,005 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. started... 2023-04-25 02:54:28,005 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. as already flushing 2023-04-25 02:54:28,005 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. 2023-04-25 02:54:28,005 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:54:28,005 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:54:28,005 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:28,005 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:54:28,005 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:54:28,006 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:28,006 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:28,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:28,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:28,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:28,007 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:28,007 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:54:28,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:28,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:28,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:28,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:28,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:28,008 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:54:28,008 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1690cd62[Count = 0] remaining members to acquire global barrier 2023-04-25 02:54:28,008 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:54:28,008 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:28,010 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:28,010 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:28,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:28,010 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:54:28,010 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:54:28,010 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:54:28,010 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:28,010 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:54:28,012 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:28,012 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:54:28,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:28,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:28,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:28,012 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:28,012 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:54:28,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:28,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:28,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:28,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:28,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:28,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:28,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:28,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:54:28,014 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:54:28,014 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:54:28,014 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:54:28,014 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:54:28,014 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:54:28,014 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:54:28,016 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:28,016 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:28,016 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:28,016 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:28,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:28,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:28,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:28,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:28,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:28,016 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:28,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:28,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:28,017 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:28,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:28,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:28,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:28,017 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:28,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:28,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:28,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:28,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:28,019 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:28,019 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:28,020 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:28,020 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:28,020 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:54:28,020 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:28,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:28,020 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:54:28,020 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:54:28,020 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:28,020 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:28,020 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:28,020 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:28,020 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:28,020 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:28,020 DEBUG [Thread-799] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:54:28,020 DEBUG [Thread-799] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:54:28,020 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:28,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:28,028 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=327 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/.tmp/C/4bd8e8ee898548f8bad0ceaa0083ac8d 2023-04-25 02:54:28,049 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/.tmp/A/084af77502304152bbfef9fec78bdc0d as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/A/084af77502304152bbfef9fec78bdc0d 2023-04-25 02:54:28,053 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/A/084af77502304152bbfef9fec78bdc0d, entries=200, sequenceid=327, filesize=32.3 K 2023-04-25 02:54:28,054 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/.tmp/B/fdb844eb97b3476999382e9f5e79839c as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/B/fdb844eb97b3476999382e9f5e79839c 2023-04-25 02:54:28,058 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/B/fdb844eb97b3476999382e9f5e79839c, entries=150, sequenceid=327, filesize=11.9 K 2023-04-25 02:54:28,059 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/.tmp/C/4bd8e8ee898548f8bad0ceaa0083ac8d as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/C/4bd8e8ee898548f8bad0ceaa0083ac8d 2023-04-25 02:54:28,062 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/C/4bd8e8ee898548f8bad0ceaa0083ac8d, entries=150, sequenceid=327, filesize=11.9 K 2023-04-25 02:54:28,063 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 388bc1373cd34a10ba8be8ddc731e198 in 879ms, sequenceid=327, compaction requested=true 2023-04-25 02:54:28,063 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 388bc1373cd34a10ba8be8ddc731e198: 2023-04-25 02:54:28,064 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 02:54:28,064 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-25 02:54:28,064 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 02:54:28,064 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-25 02:54:28,064 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 02:54:28,065 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 85758 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-25 02:54:28,065 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HStore(1912): 388bc1373cd34a10ba8be8ddc731e198/A is initiating minor compaction (all files) 2023-04-25 02:54:28,065 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37138 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-25 02:54:28,065 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 388bc1373cd34a10ba8be8ddc731e198/A in TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. 2023-04-25 02:54:28,065 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HStore(1912): 388bc1373cd34a10ba8be8ddc731e198/B is initiating minor compaction (all files) 2023-04-25 02:54:28,065 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/A/68b032bf259a4543b75456f2b17ac25a, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/A/968a15dce17541b0877f73480fa084ba, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/A/084af77502304152bbfef9fec78bdc0d] into tmpdir=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/.tmp, totalSize=83.7 K 2023-04-25 02:54:28,065 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 388bc1373cd34a10ba8be8ddc731e198/B in TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. 2023-04-25 02:54:28,065 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/B/4438dd49f6794adf822d1deab57fa16e, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/B/a72ad622ac444b9aa13ed16b5c6049fb, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/B/fdb844eb97b3476999382e9f5e79839c] into tmpdir=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/.tmp, totalSize=36.3 K 2023-04-25 02:54:28,065 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting 68b032bf259a4543b75456f2b17ac25a, keycount=150, bloomtype=ROW, size=26.0 K, encoding=NONE, compression=NONE, seqNum=285, earliestPutTs=1682391262336 2023-04-25 02:54:28,066 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting 4438dd49f6794adf822d1deab57fa16e, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=285, earliestPutTs=1682391262336 2023-04-25 02:54:28,066 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting 968a15dce17541b0877f73480fa084ba, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=314, earliestPutTs=1682391264627 2023-04-25 02:54:28,066 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting a72ad622ac444b9aa13ed16b5c6049fb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=314, earliestPutTs=1682391264627 2023-04-25 02:54:28,066 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting 084af77502304152bbfef9fec78bdc0d, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=327, earliestPutTs=1682391266759 2023-04-25 02:54:28,066 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting fdb844eb97b3476999382e9f5e79839c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=327, earliestPutTs=1682391266760 2023-04-25 02:54:28,076 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] throttle.PressureAwareThroughputController(145): 388bc1373cd34a10ba8be8ddc731e198#B#compaction#160 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-25 02:54:28,078 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] throttle.PressureAwareThroughputController(145): 388bc1373cd34a10ba8be8ddc731e198#A#compaction#159 average throughput is 3.29 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-25 02:54:28,097 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/.tmp/B/399166edb9e34453a3a2f527f9905029 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/B/399166edb9e34453a3a2f527f9905029 2023-04-25 02:54:28,101 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 388bc1373cd34a10ba8be8ddc731e198/B of 388bc1373cd34a10ba8be8ddc731e198 into 399166edb9e34453a3a2f527f9905029(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 02:54:28,102 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 388bc1373cd34a10ba8be8ddc731e198: 2023-04-25 02:54:28,102 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198., storeName=388bc1373cd34a10ba8be8ddc731e198/B, priority=13, startTime=1682391268064; duration=0sec 2023-04-25 02:54:28,102 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 02:54:28,102 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-25 02:54:28,103 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37138 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-25 02:54:28,103 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HStore(1912): 388bc1373cd34a10ba8be8ddc731e198/C is initiating minor compaction (all files) 2023-04-25 02:54:28,103 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 388bc1373cd34a10ba8be8ddc731e198/C in TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. 2023-04-25 02:54:28,103 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/C/66fc279401bf46148fb40cb78196fca4, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/C/6a2fcabbc73a4653822826c08ef3d003, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/C/4bd8e8ee898548f8bad0ceaa0083ac8d] into tmpdir=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/.tmp, totalSize=36.3 K 2023-04-25 02:54:28,103 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting 66fc279401bf46148fb40cb78196fca4, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=285, earliestPutTs=1682391262336 2023-04-25 02:54:28,104 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting 6a2fcabbc73a4653822826c08ef3d003, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=314, earliestPutTs=1682391264627 2023-04-25 02:54:28,104 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting 4bd8e8ee898548f8bad0ceaa0083ac8d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=327, earliestPutTs=1682391266760 2023-04-25 02:54:28,113 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] throttle.PressureAwareThroughputController(145): 388bc1373cd34a10ba8be8ddc731e198#C#compaction#161 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-25 02:54:28,120 DEBUG [Thread-799] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:54:28,121 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:54:28,122 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:54:28,123 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/.tmp/C/4493fcfe015045cc9e6fb5be697e4222 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/C/4493fcfe015045cc9e6fb5be697e4222 2023-04-25 02:54:28,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:54:28,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:54:28,124 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:54:28,124 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:28,124 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:54:28,124 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:54:28,124 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:28,124 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:28,126 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:28,127 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:28,127 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:28,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:28,127 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:28,127 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:54:28,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:28,128 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:28,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:54:28,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:28,128 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 388bc1373cd34a10ba8be8ddc731e198/C of 388bc1373cd34a10ba8be8ddc731e198 into 4493fcfe015045cc9e6fb5be697e4222(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 02:54:28,128 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:54:28,128 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 388bc1373cd34a10ba8be8ddc731e198: 2023-04-25 02:54:28,128 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198., storeName=388bc1373cd34a10ba8be8ddc731e198/C, priority=13, startTime=1682391268064; duration=0sec 2023-04-25 02:54:28,128 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 02:54:28,129 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:54:28,129 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:54:28,130 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:28,130 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:54:28,130 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:54:28,130 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:54:28,130 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. 2023-04-25 02:54:28,130 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. started... 2023-04-25 02:54:28,130 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 388bc1373cd34a10ba8be8ddc731e198 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-04-25 02:54:28,130 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 388bc1373cd34a10ba8be8ddc731e198, store=A 2023-04-25 02:54:28,130 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 02:54:28,130 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 388bc1373cd34a10ba8be8ddc731e198, store=B 2023-04-25 02:54:28,131 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 02:54:28,131 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 388bc1373cd34a10ba8be8ddc731e198, store=C 2023-04-25 02:54:28,131 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 02:54:28,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(9158): Flush requested on 388bc1373cd34a10ba8be8ddc731e198 2023-04-25 02:54:28,413 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. as already flushing 2023-04-25 02:54:28,427 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:28,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53610 deadline: 1682391328426, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:28,508 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/.tmp/A/1dcd836b77af48b48fe5aa61cf3dbe7e as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/A/1dcd836b77af48b48fe5aa61cf3dbe7e 2023-04-25 02:54:28,513 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 388bc1373cd34a10ba8be8ddc731e198/A of 388bc1373cd34a10ba8be8ddc731e198 into 1dcd836b77af48b48fe5aa61cf3dbe7e(size=26.1 K), total size for store is 26.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 02:54:28,513 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 388bc1373cd34a10ba8be8ddc731e198: 2023-04-25 02:54:28,513 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198., storeName=388bc1373cd34a10ba8be8ddc731e198/A, priority=13, startTime=1682391268063; duration=0sec 2023-04-25 02:54:28,513 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 02:54:28,529 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:28,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53610 deadline: 1682391328528, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:28,543 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230425c2cda85169824b2cbc04bd084b8d8d84 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425c2cda85169824b2cbc04bd084b8d8d84 2023-04-25 02:54:28,553 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=366, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/.tmp/A/b96b7b80c05c49dcb517a90d90078b41 2023-04-25 02:54:28,572 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=366 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/.tmp/B/93b64512080f46199d5a835131d9a1bf 2023-04-25 02:54:28,731 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:28,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53610 deadline: 1682391328730, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:28,885 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:28,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53664 deadline: 1682391328884, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:28,892 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:28,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53692 deadline: 1682391328891, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:28,984 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=366 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/.tmp/C/f69ffe4f9d124c66be13da18e5ad11e4 2023-04-25 02:54:28,989 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/.tmp/A/b96b7b80c05c49dcb517a90d90078b41 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/A/b96b7b80c05c49dcb517a90d90078b41 2023-04-25 02:54:28,993 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/A/b96b7b80c05c49dcb517a90d90078b41, entries=150, sequenceid=366, filesize=25.4 K 2023-04-25 02:54:28,994 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/.tmp/B/93b64512080f46199d5a835131d9a1bf as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/B/93b64512080f46199d5a835131d9a1bf 2023-04-25 02:54:29,010 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/B/93b64512080f46199d5a835131d9a1bf, entries=150, sequenceid=366, filesize=11.9 K 2023-04-25 02:54:29,012 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/.tmp/C/f69ffe4f9d124c66be13da18e5ad11e4 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/C/f69ffe4f9d124c66be13da18e5ad11e4 2023-04-25 02:54:29,017 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/C/f69ffe4f9d124c66be13da18e5ad11e4, entries=150, sequenceid=366, filesize=11.9 K 2023-04-25 02:54:29,018 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=46.96 KB/48090 for 388bc1373cd34a10ba8be8ddc731e198 in 888ms, sequenceid=366, compaction requested=false 2023-04-25 02:54:29,018 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 388bc1373cd34a10ba8be8ddc731e198: 2023-04-25 02:54:29,018 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. 2023-04-25 02:54:29,018 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:54:29,018 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:54:29,018 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:29,018 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:54:29,018 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:54:29,019 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:29,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:29,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:29,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:29,019 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:29,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:29,020 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:29,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:29,020 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:54:29,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:29,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:29,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:29,021 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:54:29,021 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@28ad6017[Count = 0] remaining members to acquire global barrier 2023-04-25 02:54:29,021 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:54:29,021 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:29,023 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:29,023 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:29,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:29,023 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:54:29,023 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:54:29,023 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:29,023 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:54:29,023 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:54:29,024 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:54:29,024 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:29,024 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:29,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:29,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:29,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:29,025 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:54:29,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:29,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:29,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:29,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:29,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:29,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:29,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:29,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:54:29,028 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:54:29,028 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:54:29,028 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:54:29,028 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:54:29,028 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:54:29,028 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:54:29,032 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:29,032 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:29,032 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:29,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:29,032 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:29,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:29,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:29,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:29,032 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:29,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:29,032 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:29,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:29,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:29,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:29,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:29,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:29,034 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:29,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:29,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:29,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:29,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:29,036 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:29,036 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:29,036 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:29,036 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:29,036 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:29,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:29,036 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:54:29,036 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:54:29,036 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:29,036 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:29,036 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:29,036 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:29,036 DEBUG [Thread-799] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:54:29,036 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:29,036 DEBUG [Thread-799] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:54:29,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:29,036 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:29,037 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:29,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(9158): Flush requested on 388bc1373cd34a10ba8be8ddc731e198 2023-04-25 02:54:29,038 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 388bc1373cd34a10ba8be8ddc731e198 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-25 02:54:29,038 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 388bc1373cd34a10ba8be8ddc731e198, store=A 2023-04-25 02:54:29,038 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:54:29,039 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 388bc1373cd34a10ba8be8ddc731e198, store=B 2023-04-25 02:54:29,039 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:54:29,039 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 388bc1373cd34a10ba8be8ddc731e198, store=C 2023-04-25 02:54:29,039 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:54:29,071 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304256e710fb3b03746518ae343e932926944 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304256e710fb3b03746518ae343e932926944 2023-04-25 02:54:29,102 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682391269087, max timestamp=1682391269100Num uniques 859; 2023-04-25 02:54:29,102 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-25 02:54:29,102 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=378, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/.tmp/A/dee9c70e403342659e7be09e2d3d230e 2023-04-25 02:54:29,105 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682391269087, max timestamp=1682391269100Num uniques 859; 2023-04-25 02:54:29,105 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682391269087, max timestamp=1682391269100Num uniques 859; 2023-04-25 02:54:29,105 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-25 02:54:29,105 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-25 02:54:29,122 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=378 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/.tmp/B/d261a845a615453ab8b9676816fdc2c3 2023-04-25 02:54:29,136 DEBUG [Thread-799] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:54:29,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:54:29,139 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:54:29,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:54:29,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:54:29,144 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:54:29,144 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:29,144 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:54:29,144 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:54:29,145 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:29,145 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:29,146 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:29,147 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:29,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:29,147 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:29,148 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:29,148 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:54:29,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:29,149 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:29,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:54:29,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:29,149 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:54:29,164 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:54:29,164 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:54:29,164 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:54:29,164 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:29,164 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:54:29,168 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:29,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 216 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53610 deadline: 1682391329166, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:29,176 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:54:29,176 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:54:29,176 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. 2023-04-25 02:54:29,176 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. started... 2023-04-25 02:54:29,177 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. as already flushing 2023-04-25 02:54:29,177 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. 2023-04-25 02:54:29,177 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:54:29,177 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:54:29,178 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:29,178 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:54:29,178 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:54:29,183 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:29,183 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:29,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:29,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:29,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:29,184 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:29,184 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:54:29,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:29,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:29,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:29,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:29,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:29,188 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:54:29,188 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5c0f63eb[Count = 0] remaining members to acquire global barrier 2023-04-25 02:54:29,188 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:54:29,188 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:29,189 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:29,189 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:29,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:29,189 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:54:29,189 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:54:29,189 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:54:29,190 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:29,190 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:54:29,191 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:54:29,191 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:29,191 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:29,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:29,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:29,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:29,191 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:54:29,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:29,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:29,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:29,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:29,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:29,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:29,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:29,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:54:29,194 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:54:29,194 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:54:29,194 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:54:29,194 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:54:29,194 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:54:29,194 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:54:29,196 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:29,196 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:29,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:29,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:29,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:29,196 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:29,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:29,196 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:29,197 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:29,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:29,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:29,197 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:29,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:29,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:29,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:29,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:29,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:29,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:29,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:29,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:29,201 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:29,204 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:29,204 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:29,204 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:29,204 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:29,204 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:29,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:29,204 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:29,204 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:29,205 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:29,205 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:29,205 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:54:29,205 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:54:29,205 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:29,205 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:29,206 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:29,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:29,207 DEBUG [Thread-799] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:54:29,207 DEBUG [Thread-799] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:54:29,272 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:29,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 218 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53610 deadline: 1682391329269, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:29,308 DEBUG [Thread-799] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:54:29,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:54:29,310 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:54:29,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:54:29,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:54:29,311 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:54:29,311 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:29,311 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:54:29,311 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:54:29,312 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:29,312 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:29,313 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:29,313 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:29,313 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:29,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:29,313 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:29,313 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:54:29,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:29,314 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:29,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:54:29,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:29,314 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:54:29,314 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:54:29,314 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:54:29,314 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:54:29,314 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:29,315 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:54:29,315 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:54:29,315 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:54:29,315 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. 2023-04-25 02:54:29,315 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. started... 2023-04-25 02:54:29,315 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. as already flushing 2023-04-25 02:54:29,315 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. 2023-04-25 02:54:29,315 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:54:29,315 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:54:29,315 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:29,315 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:54:29,315 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:54:29,317 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:29,317 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:29,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:29,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:29,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:29,317 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:29,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:29,317 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:54:29,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:29,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:29,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:29,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:29,318 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:54:29,318 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@77849567[Count = 0] remaining members to acquire global barrier 2023-04-25 02:54:29,318 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:54:29,319 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:29,320 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:29,320 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:29,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:29,320 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:54:29,320 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:54:29,320 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:54:29,320 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:29,320 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:54:29,322 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:54:29,322 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:29,322 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:54:29,322 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:29,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:29,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:29,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:29,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:29,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:29,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:29,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:29,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:29,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:29,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:29,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:54:29,324 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:54:29,324 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:54:29,325 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:54:29,325 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:54:29,325 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:54:29,325 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:54:29,326 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:29,326 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:29,326 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:29,326 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:29,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:29,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:29,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:29,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:29,327 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:29,327 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:29,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:29,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:29,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:29,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:29,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:29,328 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:29,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:29,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:29,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:29,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:29,330 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:29,330 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:29,330 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:29,330 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:29,330 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:29,330 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:29,330 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:29,330 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:29,330 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:29,330 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:29,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:29,330 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:29,330 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:29,330 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:54:29,330 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:54:29,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:29,331 DEBUG [Thread-799] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:54:29,331 DEBUG [Thread-799] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:54:29,331 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:54:29,431 DEBUG [Thread-799] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:54:29,431 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:54:29,433 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:54:29,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:54:29,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:54:29,434 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:54:29,434 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:29,434 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:54:29,434 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:54:29,435 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:29,435 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:29,436 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:29,436 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:29,436 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:29,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:29,436 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:29,436 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:54:29,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:29,437 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:29,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:54:29,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:29,437 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:54:29,437 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:54:29,437 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:54:29,437 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:54:29,437 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:29,438 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:54:29,438 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:54:29,438 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:54:29,438 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. 2023-04-25 02:54:29,438 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. started... 2023-04-25 02:54:29,438 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. as already flushing 2023-04-25 02:54:29,438 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. 2023-04-25 02:54:29,438 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:54:29,438 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:54:29,438 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:29,438 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:54:29,438 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:54:29,441 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:29,441 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:29,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:29,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:29,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:29,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:29,443 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:29,443 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:54:29,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:29,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:29,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:29,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:29,444 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:54:29,444 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@67efad28[Count = 0] remaining members to acquire global barrier 2023-04-25 02:54:29,444 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:54:29,444 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:29,445 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:29,445 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:29,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:29,445 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:54:29,445 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:54:29,445 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:29,445 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:54:29,445 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:54:29,447 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:29,447 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:54:29,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:29,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:29,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:29,447 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:29,448 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:54:29,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:29,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:29,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:29,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:29,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:29,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:29,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:29,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:54:29,450 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:54:29,450 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:54:29,450 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:54:29,450 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:54:29,450 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:54:29,450 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:54:29,451 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:29,451 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:29,451 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:29,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:29,451 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:29,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:29,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:29,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:29,451 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:29,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:29,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:29,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:29,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:29,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:29,452 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:29,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:29,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:29,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:29,453 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:29,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:29,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:29,457 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:29,457 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:29,457 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:29,457 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:54:29,457 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:54:29,457 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:29,457 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:29,457 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:29,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:29,457 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:29,457 DEBUG [Thread-799] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:54:29,457 DEBUG [Thread-799] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:54:29,457 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:29,457 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:29,457 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:29,457 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:29,457 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:29,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:29,476 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:29,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 220 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53610 deadline: 1682391329474, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:29,557 DEBUG [Thread-799] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:54:29,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:54:29,559 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:54:29,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:54:29,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:54:29,560 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:54:29,560 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:29,560 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:54:29,560 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:54:29,561 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:29,561 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:29,562 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:29,562 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:29,562 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:29,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:29,562 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:29,562 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:54:29,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:29,562 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:29,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:54:29,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:29,563 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:54:29,563 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:54:29,563 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:54:29,563 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:54:29,563 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:29,563 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:54:29,563 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:54:29,563 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:54:29,563 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. 2023-04-25 02:54:29,563 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. started... 2023-04-25 02:54:29,563 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. as already flushing 2023-04-25 02:54:29,563 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. 2023-04-25 02:54:29,563 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:54:29,563 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:54:29,563 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:29,564 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:54:29,564 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:54:29,565 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:29,565 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:29,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:29,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:29,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:29,565 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:29,565 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:54:29,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:29,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:29,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:29,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:29,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:29,567 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:54:29,567 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:54:29,567 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2481d41a[Count = 0] remaining members to acquire global barrier 2023-04-25 02:54:29,567 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:29,571 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=378 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/.tmp/C/4c0bb33453134b19a348bf4734aa994f 2023-04-25 02:54:29,572 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:29,572 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:29,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:29,572 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:54:29,572 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:54:29,572 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:54:29,572 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:29,572 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:54:29,573 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:54:29,573 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:29,573 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:29,573 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:54:29,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:29,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:29,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:29,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:29,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:29,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:29,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:29,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:29,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:29,576 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/.tmp/A/dee9c70e403342659e7be09e2d3d230e as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/A/dee9c70e403342659e7be09e2d3d230e 2023-04-25 02:54:29,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:29,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:54:29,577 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:54:29,577 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:54:29,577 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:54:29,577 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:54:29,577 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:54:29,577 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:54:29,578 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:29,578 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:29,578 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:29,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:29,578 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:29,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:29,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:29,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:29,579 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:29,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:29,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:29,579 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:29,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:29,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:29,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:29,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:29,581 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/A/dee9c70e403342659e7be09e2d3d230e, entries=200, sequenceid=378, filesize=32.3 K 2023-04-25 02:54:29,581 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:29,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:29,582 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/.tmp/B/d261a845a615453ab8b9676816fdc2c3 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/B/d261a845a615453ab8b9676816fdc2c3 2023-04-25 02:54:29,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:29,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:29,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:29,584 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:29,584 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:29,584 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:29,584 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:29,584 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:29,584 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:29,584 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:29,584 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:54:29,584 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:54:29,585 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:54:29,584 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:29,585 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:29,585 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:29,585 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:29,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:29,585 DEBUG [Thread-799] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:54:29,585 DEBUG [Thread-799] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:54:29,585 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:29,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:29,586 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/B/d261a845a615453ab8b9676816fdc2c3, entries=150, sequenceid=378, filesize=11.9 K 2023-04-25 02:54:29,588 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/.tmp/C/4c0bb33453134b19a348bf4734aa994f as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/C/4c0bb33453134b19a348bf4734aa994f 2023-04-25 02:54:29,591 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/C/4c0bb33453134b19a348bf4734aa994f, entries=150, sequenceid=378, filesize=11.9 K 2023-04-25 02:54:29,592 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=129.71 KB/132820 for 388bc1373cd34a10ba8be8ddc731e198 in 554ms, sequenceid=378, compaction requested=true 2023-04-25 02:54:29,592 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 388bc1373cd34a10ba8be8ddc731e198: 2023-04-25 02:54:29,592 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 02:54:29,592 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 02:54:29,592 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-25 02:54:29,592 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-25 02:54:29,592 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 02:54:29,597 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37240 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-25 02:54:29,597 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 85860 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-25 02:54:29,597 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HStore(1912): 388bc1373cd34a10ba8be8ddc731e198/B is initiating minor compaction (all files) 2023-04-25 02:54:29,597 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HStore(1912): 388bc1373cd34a10ba8be8ddc731e198/A is initiating minor compaction (all files) 2023-04-25 02:54:29,597 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 388bc1373cd34a10ba8be8ddc731e198/B in TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. 2023-04-25 02:54:29,597 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 388bc1373cd34a10ba8be8ddc731e198/A in TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. 2023-04-25 02:54:29,597 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/B/399166edb9e34453a3a2f527f9905029, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/B/93b64512080f46199d5a835131d9a1bf, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/B/d261a845a615453ab8b9676816fdc2c3] into tmpdir=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/.tmp, totalSize=36.4 K 2023-04-25 02:54:29,597 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/A/1dcd836b77af48b48fe5aa61cf3dbe7e, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/A/b96b7b80c05c49dcb517a90d90078b41, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/A/dee9c70e403342659e7be09e2d3d230e] into tmpdir=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/.tmp, totalSize=83.8 K 2023-04-25 02:54:29,598 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting 399166edb9e34453a3a2f527f9905029, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=327, earliestPutTs=1682391266760 2023-04-25 02:54:29,598 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting 1dcd836b77af48b48fe5aa61cf3dbe7e, keycount=150, bloomtype=ROW, size=26.1 K, encoding=NONE, compression=NONE, seqNum=327, earliestPutTs=1682391266760 2023-04-25 02:54:29,598 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting 93b64512080f46199d5a835131d9a1bf, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=366, earliestPutTs=1682391267286 2023-04-25 02:54:29,598 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting b96b7b80c05c49dcb517a90d90078b41, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=366, earliestPutTs=1682391267286 2023-04-25 02:54:29,599 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting d261a845a615453ab8b9676816fdc2c3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=378, earliestPutTs=1682391268422 2023-04-25 02:54:29,599 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting dee9c70e403342659e7be09e2d3d230e, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=378, earliestPutTs=1682391268412 2023-04-25 02:54:29,617 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] throttle.PressureAwareThroughputController(145): 388bc1373cd34a10ba8be8ddc731e198#B#compaction#168 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-25 02:54:29,629 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] throttle.PressureAwareThroughputController(145): 388bc1373cd34a10ba8be8ddc731e198#A#compaction#169 average throughput is 1.79 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-25 02:54:29,685 DEBUG [Thread-799] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:54:29,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:54:29,687 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:54:29,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:54:29,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:54:29,689 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:54:29,689 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:29,689 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:54:29,689 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:54:29,689 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:29,689 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:29,690 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:29,690 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:29,690 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:29,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:29,690 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:29,691 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:54:29,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:29,691 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:29,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:54:29,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:29,691 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:54:29,692 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:54:29,692 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:54:29,692 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:54:29,692 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:29,692 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:54:29,700 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:54:29,700 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:54:29,700 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. 2023-04-25 02:54:29,700 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. started... 2023-04-25 02:54:29,700 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 388bc1373cd34a10ba8be8ddc731e198 3/3 column families, dataSize=129.71 KB heapSize=370.14 KB 2023-04-25 02:54:29,700 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 388bc1373cd34a10ba8be8ddc731e198, store=A 2023-04-25 02:54:29,700 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 02:54:29,700 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 388bc1373cd34a10ba8be8ddc731e198, store=B 2023-04-25 02:54:29,700 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 02:54:29,700 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 388bc1373cd34a10ba8be8ddc731e198, store=C 2023-04-25 02:54:29,701 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 02:54:29,730 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304257f76aa0f7efd4409861200c8d48b9805 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304257f76aa0f7efd4409861200c8d48b9805 2023-04-25 02:54:29,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(9158): Flush requested on 388bc1373cd34a10ba8be8ddc731e198 2023-04-25 02:54:29,781 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. as already flushing 2023-04-25 02:54:29,810 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:29,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 230 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53610 deadline: 1682391329803, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:29,915 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:29,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 232 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53610 deadline: 1682391329911, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:30,054 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/.tmp/B/eb3e71697d8a4fea92a7fcd8d60f15f6 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/B/eb3e71697d8a4fea92a7fcd8d60f15f6 2023-04-25 02:54:30,064 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 388bc1373cd34a10ba8be8ddc731e198/B of 388bc1373cd34a10ba8be8ddc731e198 into eb3e71697d8a4fea92a7fcd8d60f15f6(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 02:54:30,065 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 388bc1373cd34a10ba8be8ddc731e198: 2023-04-25 02:54:30,065 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198., storeName=388bc1373cd34a10ba8be8ddc731e198/B, priority=13, startTime=1682391269592; duration=0sec 2023-04-25 02:54:30,065 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 02:54:30,065 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-25 02:54:30,066 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37240 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-25 02:54:30,066 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HStore(1912): 388bc1373cd34a10ba8be8ddc731e198/C is initiating minor compaction (all files) 2023-04-25 02:54:30,066 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 388bc1373cd34a10ba8be8ddc731e198/C in TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. 2023-04-25 02:54:30,066 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/C/4493fcfe015045cc9e6fb5be697e4222, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/C/f69ffe4f9d124c66be13da18e5ad11e4, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/C/4c0bb33453134b19a348bf4734aa994f] into tmpdir=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/.tmp, totalSize=36.4 K 2023-04-25 02:54:30,067 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting 4493fcfe015045cc9e6fb5be697e4222, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=327, earliestPutTs=1682391266760 2023-04-25 02:54:30,067 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting f69ffe4f9d124c66be13da18e5ad11e4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=366, earliestPutTs=1682391267286 2023-04-25 02:54:30,068 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting 4c0bb33453134b19a348bf4734aa994f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=378, earliestPutTs=1682391268422 2023-04-25 02:54:30,077 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] throttle.PressureAwareThroughputController(145): 388bc1373cd34a10ba8be8ddc731e198#C#compaction#171 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-25 02:54:30,080 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/.tmp/A/d0e06dbb28f64ccb9494d2dd74264e4b as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/A/d0e06dbb28f64ccb9494d2dd74264e4b 2023-04-25 02:54:30,086 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 388bc1373cd34a10ba8be8ddc731e198/A of 388bc1373cd34a10ba8be8ddc731e198 into d0e06dbb28f64ccb9494d2dd74264e4b(size=26.2 K), total size for store is 26.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 02:54:30,086 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 388bc1373cd34a10ba8be8ddc731e198: 2023-04-25 02:54:30,086 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198., storeName=388bc1373cd34a10ba8be8ddc731e198/A, priority=13, startTime=1682391269592; duration=0sec 2023-04-25 02:54:30,086 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 02:54:30,100 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/.tmp/C/5ebf3082bbe944acb544ab7ad05afb93 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/C/5ebf3082bbe944acb544ab7ad05afb93 2023-04-25 02:54:30,105 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 388bc1373cd34a10ba8be8ddc731e198/C of 388bc1373cd34a10ba8be8ddc731e198 into 5ebf3082bbe944acb544ab7ad05afb93(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 02:54:30,105 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 388bc1373cd34a10ba8be8ddc731e198: 2023-04-25 02:54:30,105 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198., storeName=388bc1373cd34a10ba8be8ddc731e198/C, priority=13, startTime=1682391269592; duration=0sec 2023-04-25 02:54:30,105 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 02:54:30,118 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:30,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 234 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53610 deadline: 1682391330117, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:30,145 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=414, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/.tmp/A/51a2226ab5e944d9847136275506a7dd 2023-04-25 02:54:30,424 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:30,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 236 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53610 deadline: 1682391330421, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:30,561 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=414 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/.tmp/B/f61be2cd26f549a69ce1eb9593f8445a 2023-04-25 02:54:30,591 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=414 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/.tmp/C/9f1df4a42e1d462c8d63fcbc975388a5 2023-04-25 02:54:30,604 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/.tmp/A/51a2226ab5e944d9847136275506a7dd as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/A/51a2226ab5e944d9847136275506a7dd 2023-04-25 02:54:30,609 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/A/51a2226ab5e944d9847136275506a7dd, entries=150, sequenceid=414, filesize=25.4 K 2023-04-25 02:54:30,610 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/.tmp/B/f61be2cd26f549a69ce1eb9593f8445a as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/B/f61be2cd26f549a69ce1eb9593f8445a 2023-04-25 02:54:30,615 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/B/f61be2cd26f549a69ce1eb9593f8445a, entries=150, sequenceid=414, filesize=11.9 K 2023-04-25 02:54:30,616 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/.tmp/C/9f1df4a42e1d462c8d63fcbc975388a5 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/C/9f1df4a42e1d462c8d63fcbc975388a5 2023-04-25 02:54:30,621 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/C/9f1df4a42e1d462c8d63fcbc975388a5, entries=150, sequenceid=414, filesize=11.9 K 2023-04-25 02:54:30,622 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~129.71 KB/132820, heapSize ~370.09 KB/378976, currentSize=53.67 KB/54960 for 388bc1373cd34a10ba8be8ddc731e198 in 922ms, sequenceid=414, compaction requested=false 2023-04-25 02:54:30,622 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 388bc1373cd34a10ba8be8ddc731e198: 2023-04-25 02:54:30,622 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. 2023-04-25 02:54:30,622 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:54:30,622 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:54:30,622 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:30,623 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:54:30,623 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:54:30,624 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:30,624 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:30,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:30,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:30,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:30,628 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:30,628 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:54:30,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:30,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:30,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:30,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:30,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:30,631 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:54:30,631 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3b6eb5a[Count = 0] remaining members to acquire global barrier 2023-04-25 02:54:30,631 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:54:30,631 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:30,633 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:30,633 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:30,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:30,633 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:54:30,633 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:54:30,633 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:54:30,635 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:30,635 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:54:30,635 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:30,635 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:54:30,636 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:54:30,636 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:54:30,636 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:54:30,636 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:54:30,636 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:54:30,636 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:54:30,636 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:54:30,637 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:30,637 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:30,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:30,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:30,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:30,637 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:30,637 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:30,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:30,638 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:30,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:30,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:30,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:30,638 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:30,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:30,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:30,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:30,640 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:30,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:30,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:30,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:30,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:30,644 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:30,644 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:30,644 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:30,644 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:30,644 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:30,644 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:30,644 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:30,644 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:30,644 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:30,644 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:54:30,644 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:54:30,644 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:30,644 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:30,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:30,644 DEBUG [Thread-799] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:54:30,644 DEBUG [Thread-799] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:54:30,645 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:30,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:30,745 DEBUG [Thread-799] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:54:30,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:54:30,747 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:54:30,748 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:54:30,748 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:54:30,748 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:54:30,748 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:30,748 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:54:30,748 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:54:30,748 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:30,748 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:30,750 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:30,750 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:30,750 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:30,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:30,750 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:30,750 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:54:30,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:30,750 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:30,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:54:30,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:30,751 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:54:30,751 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:54:30,751 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:54:30,751 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:54:30,751 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:30,751 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:54:30,752 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:54:30,752 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:54:30,752 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. 2023-04-25 02:54:30,752 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. started... 2023-04-25 02:54:30,752 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 388bc1373cd34a10ba8be8ddc731e198 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-25 02:54:30,752 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 388bc1373cd34a10ba8be8ddc731e198, store=A 2023-04-25 02:54:30,752 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:54:30,752 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 388bc1373cd34a10ba8be8ddc731e198, store=B 2023-04-25 02:54:30,752 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:54:30,752 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 388bc1373cd34a10ba8be8ddc731e198, store=C 2023-04-25 02:54:30,752 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:54:30,774 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230425b25966fba6f24010a6f988c9625d55f4 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425b25966fba6f24010a6f988c9625d55f4 2023-04-25 02:54:30,786 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=428, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/.tmp/A/98ed6608a8684edba57ea59d209f0bfd 2023-04-25 02:54:30,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(9158): Flush requested on 388bc1373cd34a10ba8be8ddc731e198 2023-04-25 02:54:30,898 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. as already flushing 2023-04-25 02:54:30,953 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:30,953 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53692 deadline: 1682391330951, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:30,954 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682391270947, max timestamp=1682391270952Num uniques 859; 2023-04-25 02:54:30,954 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682391270947, max timestamp=1682391270952Num uniques 859; 2023-04-25 02:54:30,954 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682391270947, max timestamp=1682391270952Num uniques 859; 2023-04-25 02:54:30,954 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-25 02:54:30,954 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-25 02:54:30,954 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-25 02:54:30,982 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:30,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 247 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53610 deadline: 1682391330976, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:30,983 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:30,983 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53664 deadline: 1682391330977, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:31,056 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:31,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53692 deadline: 1682391331055, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:31,086 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:31,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53664 deadline: 1682391331083, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:31,086 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:31,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 249 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53610 deadline: 1682391331084, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:31,206 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=428 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/.tmp/B/856ada7d982948a0a9d62cbd92880571 2023-04-25 02:54:31,259 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:31,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53692 deadline: 1682391331257, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:31,288 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:31,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53664 deadline: 1682391331288, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:31,289 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:31,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 251 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53610 deadline: 1682391331288, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:31,565 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:31,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53692 deadline: 1682391331562, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:31,591 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:31,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53664 deadline: 1682391331589, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:31,591 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:31,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 253 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53610 deadline: 1682391331590, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:31,618 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=428 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/.tmp/C/eed598aa6b89408dabbb06f0bff30e06 2023-04-25 02:54:31,623 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/.tmp/A/98ed6608a8684edba57ea59d209f0bfd as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/A/98ed6608a8684edba57ea59d209f0bfd 2023-04-25 02:54:31,628 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/A/98ed6608a8684edba57ea59d209f0bfd, entries=150, sequenceid=428, filesize=25.4 K 2023-04-25 02:54:31,631 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/.tmp/B/856ada7d982948a0a9d62cbd92880571 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/B/856ada7d982948a0a9d62cbd92880571 2023-04-25 02:54:31,635 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/B/856ada7d982948a0a9d62cbd92880571, entries=150, sequenceid=428, filesize=11.9 K 2023-04-25 02:54:31,636 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/.tmp/C/eed598aa6b89408dabbb06f0bff30e06 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/C/eed598aa6b89408dabbb06f0bff30e06 2023-04-25 02:54:31,640 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/C/eed598aa6b89408dabbb06f0bff30e06, entries=150, sequenceid=428, filesize=11.9 K 2023-04-25 02:54:31,641 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 388bc1373cd34a10ba8be8ddc731e198 in 889ms, sequenceid=428, compaction requested=true 2023-04-25 02:54:31,641 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 388bc1373cd34a10ba8be8ddc731e198: 2023-04-25 02:54:31,641 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. 2023-04-25 02:54:31,641 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:54:31,641 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:54:31,641 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:31,641 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:54:31,641 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:54:31,645 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:31,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:31,645 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:31,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:31,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:31,645 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:31,646 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:54:31,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:31,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:31,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:31,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:31,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:31,647 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:54:31,647 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@504a79e6[Count = 0] remaining members to acquire global barrier 2023-04-25 02:54:31,647 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:54:31,647 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:31,648 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:31,648 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:31,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:31,648 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:54:31,648 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:54:31,648 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:54:31,648 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:31,648 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:54:31,651 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:31,651 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:54:31,651 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:31,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:31,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:31,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:31,651 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:54:31,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:31,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:31,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:31,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:31,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:31,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:31,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:31,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:54:31,653 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:54:31,653 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:54:31,653 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:54:31,653 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:54:31,653 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:54:31,653 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:54:31,655 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:31,655 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:31,655 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:31,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:31,655 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:31,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:31,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:31,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:31,655 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:31,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:31,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:31,655 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:31,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:31,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:31,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:31,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:31,656 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:31,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:31,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:31,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:31,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:31,660 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:31,660 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:31,660 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:31,660 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:31,660 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:31,660 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:31,660 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:31,660 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:31,660 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:31,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:31,660 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:54:31,660 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:54:31,660 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:31,660 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:31,661 DEBUG [Thread-799] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:54:31,661 DEBUG [Thread-799] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:54:31,661 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:31,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:31,761 DEBUG [Thread-799] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:54:31,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:54:31,763 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:54:31,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:54:31,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:54:31,764 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:54:31,764 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:31,764 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:54:31,764 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:54:31,765 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:31,765 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:31,766 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:31,766 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:31,766 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:31,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:31,766 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:31,766 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:54:31,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:31,767 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:31,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:54:31,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:31,767 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:54:31,767 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:54:31,767 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:54:31,767 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:54:31,767 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:31,768 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:54:31,768 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:54:31,768 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:54:31,768 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. 2023-04-25 02:54:31,768 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. started... 2023-04-25 02:54:31,768 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 388bc1373cd34a10ba8be8ddc731e198 3/3 column families, dataSize=127.47 KB heapSize=355.69 KB 2023-04-25 02:54:31,768 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 388bc1373cd34a10ba8be8ddc731e198, store=A 2023-04-25 02:54:31,768 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 02:54:31,768 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 388bc1373cd34a10ba8be8ddc731e198, store=B 2023-04-25 02:54:31,768 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 02:54:31,768 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 388bc1373cd34a10ba8be8ddc731e198, store=C 2023-04-25 02:54:31,768 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 02:54:31,787 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023042556232c6992ab4bfdaab910ea58c94ba0 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042556232c6992ab4bfdaab910ea58c94ba0 2023-04-25 02:54:31,798 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=462, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/.tmp/A/37060278023d46adae9b2384f3071028 2023-04-25 02:54:31,813 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=462 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/.tmp/B/ae9982eb13a84fd4adfd2687f6745d33 2023-04-25 02:54:32,035 DEBUG [Thread-806] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x42d325f4 to 127.0.0.1:62054 2023-04-25 02:54:32,035 DEBUG [Thread-806] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-25 02:54:32,036 DEBUG [Thread-802] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4a0fd7b4 to 127.0.0.1:62054 2023-04-25 02:54:32,036 DEBUG [Thread-800] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x421e5356 to 127.0.0.1:62054 2023-04-25 02:54:32,036 DEBUG [Thread-802] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-25 02:54:32,036 DEBUG [Thread-800] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-25 02:54:32,039 DEBUG [Thread-804] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1a7b66a3 to 127.0.0.1:62054 2023-04-25 02:54:32,039 DEBUG [Thread-804] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-25 02:54:32,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(9158): Flush requested on 388bc1373cd34a10ba8be8ddc731e198 2023-04-25 02:54:32,068 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. as already flushing 2023-04-25 02:54:32,068 DEBUG [Thread-795] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x210d1008 to 127.0.0.1:62054 2023-04-25 02:54:32,068 DEBUG [Thread-795] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-25 02:54:32,093 DEBUG [Thread-791] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x24bd3872 to 127.0.0.1:62054 2023-04-25 02:54:32,094 DEBUG [Thread-791] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-25 02:54:32,097 DEBUG [Thread-797] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x00c2598b to 127.0.0.1:62054 2023-04-25 02:54:32,098 DEBUG [Thread-797] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-25 02:54:32,234 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=462 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/.tmp/C/8385df4aab19453d94a0f06461e796a1 2023-04-25 02:54:32,240 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/.tmp/A/37060278023d46adae9b2384f3071028 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/A/37060278023d46adae9b2384f3071028 2023-04-25 02:54:32,243 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/A/37060278023d46adae9b2384f3071028, entries=150, sequenceid=462, filesize=25.4 K 2023-04-25 02:54:32,244 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/.tmp/B/ae9982eb13a84fd4adfd2687f6745d33 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/B/ae9982eb13a84fd4adfd2687f6745d33 2023-04-25 02:54:32,247 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/B/ae9982eb13a84fd4adfd2687f6745d33, entries=150, sequenceid=462, filesize=11.9 K 2023-04-25 02:54:32,248 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/.tmp/C/8385df4aab19453d94a0f06461e796a1 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/C/8385df4aab19453d94a0f06461e796a1 2023-04-25 02:54:32,253 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/C/8385df4aab19453d94a0f06461e796a1, entries=150, sequenceid=462, filesize=11.9 K 2023-04-25 02:54:32,253 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~355.64 KB/364176, currentSize=20.13 KB/20610 for 388bc1373cd34a10ba8be8ddc731e198 in 485ms, sequenceid=462, compaction requested=true 2023-04-25 02:54:32,253 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 388bc1373cd34a10ba8be8ddc731e198: 2023-04-25 02:54:32,253 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. 2023-04-25 02:54:32,253 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:54:32,253 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:54:32,253 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:32,254 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:54:32,254 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:54:32,255 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:32,255 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:32,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:32,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:32,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:32,255 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:32,255 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:54:32,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:32,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:32,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:32,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:32,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:32,256 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:54:32,257 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4ad6182a[Count = 0] remaining members to acquire global barrier 2023-04-25 02:54:32,257 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:54:32,257 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:32,261 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:32,261 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:32,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:32,261 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:54:32,261 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:54:32,261 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:32,261 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:54:32,261 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:54:32,263 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:54:32,263 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:32,263 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:32,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:32,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:32,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:32,263 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:54:32,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:32,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:32,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:32,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:32,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:32,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:32,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:32,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:54:32,265 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:54:32,265 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:54:32,265 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:54:32,265 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:54:32,265 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:54:32,265 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:54:32,267 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:32,267 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:32,267 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:32,267 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:32,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:32,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:32,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:32,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:32,267 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:32,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:32,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:32,267 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:32,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:32,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:32,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:32,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:32,272 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:32,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:32,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:32,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:32,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:32,274 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:32,274 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:32,274 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:32,274 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:32,274 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:54:32,274 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:54:32,274 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:32,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:32,274 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:32,274 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:32,275 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:32,275 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:32,275 DEBUG [Thread-799] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:54:32,275 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:32,275 DEBUG [Thread-799] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:54:32,275 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:32,275 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:32,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:32,375 DEBUG [Thread-799] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:54:32,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:54:33,242 DEBUG [Thread-789] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x55cbb115 to 127.0.0.1:62054 2023-04-25 02:54:33,243 DEBUG [Thread-789] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-25 02:54:33,281 DEBUG [Thread-793] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7aba8e46 to 127.0.0.1:62054 2023-04-25 02:54:33,281 DEBUG [Thread-793] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-25 02:54:33,281 INFO [Listener at localhost/41915] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-25 02:54:33,281 INFO [Listener at localhost/41915] hbase.AcidGuaranteesTestTool(394): wrote 28 2023-04-25 02:54:33,282 INFO [Listener at localhost/41915] hbase.AcidGuaranteesTestTool(394): wrote 92 2023-04-25 02:54:33,282 INFO [Listener at localhost/41915] hbase.AcidGuaranteesTestTool(394): wrote 25 2023-04-25 02:54:33,282 INFO [Listener at localhost/41915] hbase.AcidGuaranteesTestTool(394): wrote 79 2023-04-25 02:54:33,282 INFO [Listener at localhost/41915] hbase.AcidGuaranteesTestTool(394): wrote 156 2023-04-25 02:54:33,282 INFO [Listener at localhost/41915] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-25 02:54:33,282 INFO [Listener at localhost/41915] hbase.AcidGuaranteesTestTool(398): read 5089 2023-04-25 02:54:33,282 INFO [Listener at localhost/41915] hbase.AcidGuaranteesTestTool(398): read 5123 2023-04-25 02:54:33,282 INFO [Listener at localhost/41915] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-25 02:54:33,282 INFO [Listener at localhost/41915] hbase.AcidGuaranteesTestTool(402): scanned 2192 2023-04-25 02:54:33,282 INFO [Listener at localhost/41915] hbase.AcidGuaranteesTestTool(403): verified 6569 rows 2023-04-25 02:54:33,282 INFO [Listener at localhost/41915] hbase.AcidGuaranteesTestTool(402): scanned 2174 2023-04-25 02:54:33,282 INFO [Listener at localhost/41915] hbase.AcidGuaranteesTestTool(403): verified 6521 rows 2023-04-25 02:54:33,282 INFO [Listener at localhost/41915] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-25 02:54:33,282 DEBUG [Listener at localhost/41915] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x324dc814 to 127.0.0.1:62054 2023-04-25 02:54:33,282 DEBUG [Listener at localhost/41915] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-25 02:54:33,283 INFO [Listener at localhost/41915] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-25 02:54:33,284 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-04-25 02:54:33,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure2.ProcedureExecutor(1029): Stored pid=24, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-25 02:54:33,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-25 02:54:33,288 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682391273288"}]},"ts":"1682391273288"} 2023-04-25 02:54:33,289 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-25 02:54:33,291 INFO [PEWorker-1] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-25 02:54:33,292 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=25, ppid=24, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=388bc1373cd34a10ba8be8ddc731e198, UNASSIGN}] 2023-04-25 02:54:33,293 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=25, ppid=24, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=388bc1373cd34a10ba8be8ddc731e198, UNASSIGN 2023-04-25 02:54:33,294 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=25 updating hbase:meta row=388bc1373cd34a10ba8be8ddc731e198, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:33,294 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682391273294"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682391273294"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682391273294"}]},"ts":"1682391273294"} 2023-04-25 02:54:33,295 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=26, ppid=25, state=RUNNABLE; CloseRegionProcedure 388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023}] 2023-04-25 02:54:33,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-25 02:54:33,446 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 388bc1373cd34a10ba8be8ddc731e198 2023-04-25 02:54:33,447 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 388bc1373cd34a10ba8be8ddc731e198, disabling compactions & flushes 2023-04-25 02:54:33,447 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. 2023-04-25 02:54:33,447 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. 2023-04-25 02:54:33,447 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. after waiting 0 ms 2023-04-25 02:54:33,447 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. 2023-04-25 02:54:33,447 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 388bc1373cd34a10ba8be8ddc731e198 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-04-25 02:54:33,447 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 388bc1373cd34a10ba8be8ddc731e198, store=A 2023-04-25 02:54:33,447 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:54:33,447 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 388bc1373cd34a10ba8be8ddc731e198, store=B 2023-04-25 02:54:33,447 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:54:33,447 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 388bc1373cd34a10ba8be8ddc731e198, store=C 2023-04-25 02:54:33,447 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:54:33,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-25 02:54:33,859 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230425e4bfb13b205540529c65499ae8911d24 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425e4bfb13b205540529c65499ae8911d24 2023-04-25 02:54:33,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-25 02:54:34,264 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=470, memsize=11.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/.tmp/A/70652f454fa54e06a6943c7dc7ddaa3c 2023-04-25 02:54:34,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-25 02:54:34,674 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=470 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/.tmp/B/22be35f2887e4d859f2cc2785dd1a1fd 2023-04-25 02:54:35,084 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=470 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/.tmp/C/05b5e3d7e1064f4584a13593b6714c29 2023-04-25 02:54:35,088 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/.tmp/A/70652f454fa54e06a6943c7dc7ddaa3c as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/A/70652f454fa54e06a6943c7dc7ddaa3c 2023-04-25 02:54:35,092 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/A/70652f454fa54e06a6943c7dc7ddaa3c, entries=150, sequenceid=470, filesize=25.4 K 2023-04-25 02:54:35,092 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/.tmp/B/22be35f2887e4d859f2cc2785dd1a1fd as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/B/22be35f2887e4d859f2cc2785dd1a1fd 2023-04-25 02:54:35,096 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/B/22be35f2887e4d859f2cc2785dd1a1fd, entries=150, sequenceid=470, filesize=11.9 K 2023-04-25 02:54:35,097 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/.tmp/C/05b5e3d7e1064f4584a13593b6714c29 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/C/05b5e3d7e1064f4584a13593b6714c29 2023-04-25 02:54:35,100 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/C/05b5e3d7e1064f4584a13593b6714c29, entries=150, sequenceid=470, filesize=11.9 K 2023-04-25 02:54:35,101 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=0 B/0 for 388bc1373cd34a10ba8be8ddc731e198 in 1654ms, sequenceid=470, compaction requested=true 2023-04-25 02:54:35,123 DEBUG [StoreCloser-TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/A/526ce0eef8a04f3a8c1b08316b8dc391, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/A/25246c3480d543cea2f7d1c2b9eb48f8, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/A/7c2d5bcd4cbd44eb9aa87771ea9ec9ac, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/A/849a15cde2b844a38357800035099e73, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/A/6b838f77696840df9ac1aafb538daec5, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/A/4a74096ea9a449fd99fa95ee4c6f0e9b, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/A/71e2cc2867584dd4900be2b96d734d12, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/A/c26090aab11042a7a3676ac917f3de43, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/A/aca01b8fefaf426083704570f714eb1a, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/A/9e04451bc8684802896548ec0b573174, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/A/2eea0cf801f9417594e3f3f0a6dd3c7e, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/A/0f7249ffa7844ca7adae49a2cd57a4f6, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/A/7081619b390a44a9b79a1aa91b66d7d6, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/A/044c787b709c4d3eaf1b670e394639be, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/A/f9a9473b993647ab9c4f0bf85254b463, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/A/2c1b6a186a05455eae6e2bb8021b7ad7, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/A/3232e5011cbf44579e5238a7576bb911, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/A/68b032bf259a4543b75456f2b17ac25a, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/A/968a15dce17541b0877f73480fa084ba, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/A/084af77502304152bbfef9fec78bdc0d, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/A/1dcd836b77af48b48fe5aa61cf3dbe7e, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/A/b96b7b80c05c49dcb517a90d90078b41, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/A/dee9c70e403342659e7be09e2d3d230e] to archive 2023-04-25 02:54:35,124 DEBUG [StoreCloser-TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-25 02:54:35,126 DEBUG [StoreCloser-TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/A/526ce0eef8a04f3a8c1b08316b8dc391 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/A/526ce0eef8a04f3a8c1b08316b8dc391 2023-04-25 02:54:35,127 DEBUG [StoreCloser-TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/A/25246c3480d543cea2f7d1c2b9eb48f8 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/A/25246c3480d543cea2f7d1c2b9eb48f8 2023-04-25 02:54:35,128 DEBUG [StoreCloser-TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/A/7c2d5bcd4cbd44eb9aa87771ea9ec9ac to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/A/7c2d5bcd4cbd44eb9aa87771ea9ec9ac 2023-04-25 02:54:35,130 DEBUG [StoreCloser-TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/A/849a15cde2b844a38357800035099e73 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/A/849a15cde2b844a38357800035099e73 2023-04-25 02:54:35,131 DEBUG [StoreCloser-TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/A/6b838f77696840df9ac1aafb538daec5 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/A/6b838f77696840df9ac1aafb538daec5 2023-04-25 02:54:35,132 DEBUG [StoreCloser-TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/A/4a74096ea9a449fd99fa95ee4c6f0e9b to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/A/4a74096ea9a449fd99fa95ee4c6f0e9b 2023-04-25 02:54:35,133 DEBUG [StoreCloser-TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/A/71e2cc2867584dd4900be2b96d734d12 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/A/71e2cc2867584dd4900be2b96d734d12 2023-04-25 02:54:35,134 DEBUG [StoreCloser-TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/A/c26090aab11042a7a3676ac917f3de43 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/A/c26090aab11042a7a3676ac917f3de43 2023-04-25 02:54:35,135 DEBUG [StoreCloser-TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/A/aca01b8fefaf426083704570f714eb1a to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/A/aca01b8fefaf426083704570f714eb1a 2023-04-25 02:54:35,136 DEBUG [StoreCloser-TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/A/9e04451bc8684802896548ec0b573174 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/A/9e04451bc8684802896548ec0b573174 2023-04-25 02:54:35,137 DEBUG [StoreCloser-TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/A/2eea0cf801f9417594e3f3f0a6dd3c7e to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/A/2eea0cf801f9417594e3f3f0a6dd3c7e 2023-04-25 02:54:35,138 DEBUG [StoreCloser-TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/A/0f7249ffa7844ca7adae49a2cd57a4f6 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/A/0f7249ffa7844ca7adae49a2cd57a4f6 2023-04-25 02:54:35,139 DEBUG [StoreCloser-TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/A/7081619b390a44a9b79a1aa91b66d7d6 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/A/7081619b390a44a9b79a1aa91b66d7d6 2023-04-25 02:54:35,140 DEBUG [StoreCloser-TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/A/044c787b709c4d3eaf1b670e394639be to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/A/044c787b709c4d3eaf1b670e394639be 2023-04-25 02:54:35,141 DEBUG [StoreCloser-TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/A/f9a9473b993647ab9c4f0bf85254b463 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/A/f9a9473b993647ab9c4f0bf85254b463 2023-04-25 02:54:35,142 DEBUG [StoreCloser-TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/A/2c1b6a186a05455eae6e2bb8021b7ad7 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/A/2c1b6a186a05455eae6e2bb8021b7ad7 2023-04-25 02:54:35,143 DEBUG [StoreCloser-TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/A/3232e5011cbf44579e5238a7576bb911 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/A/3232e5011cbf44579e5238a7576bb911 2023-04-25 02:54:35,145 DEBUG [StoreCloser-TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/A/68b032bf259a4543b75456f2b17ac25a to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/A/68b032bf259a4543b75456f2b17ac25a 2023-04-25 02:54:35,145 DEBUG [StoreCloser-TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/A/968a15dce17541b0877f73480fa084ba to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/A/968a15dce17541b0877f73480fa084ba 2023-04-25 02:54:35,146 DEBUG [StoreCloser-TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/A/084af77502304152bbfef9fec78bdc0d to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/A/084af77502304152bbfef9fec78bdc0d 2023-04-25 02:54:35,147 DEBUG [StoreCloser-TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/A/1dcd836b77af48b48fe5aa61cf3dbe7e to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/A/1dcd836b77af48b48fe5aa61cf3dbe7e 2023-04-25 02:54:35,148 DEBUG [StoreCloser-TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/A/b96b7b80c05c49dcb517a90d90078b41 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/A/b96b7b80c05c49dcb517a90d90078b41 2023-04-25 02:54:35,149 DEBUG [StoreCloser-TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/A/dee9c70e403342659e7be09e2d3d230e to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/A/dee9c70e403342659e7be09e2d3d230e 2023-04-25 02:54:35,152 DEBUG [StoreCloser-TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/B/9d672d5b816b4f128fd34506fb1fa281, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/B/e411495647fd4bfe994149ae41b61d23, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/B/2500b86ed0524a3488809f7eb6811311, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/B/ad7625f91b8645ad838990f1acc5878e, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/B/e2f4d75feec94c57b6eb0c32ef4fb5fa, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/B/8b574ddc194a4d29b92fcf273e5ce506, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/B/67f93a6b068a4cde974592dae873f4b3, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/B/bcce52a27cb64fca8262217a8ec2c06e, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/B/c40efdd9c0884675ba292c302497098f, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/B/522cb9c8f84b426181ee9bb7dc9fc27e, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/B/314f2f0378cb41f58618a457bc84285f, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/B/0ab45ceb5c454b62b269dd7d54345b64, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/B/4c8aee74a42f4267aae1cfc5c4096dc6, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/B/55550be5974a4c6c82dc75370e638ac3, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/B/107b8acc22fb42ec84647cfd2033d076, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/B/5e8755f5eada409bb76a1b107a3795a3, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/B/4438dd49f6794adf822d1deab57fa16e, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/B/0f07c3654503421995bc37dea0c599c9, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/B/a72ad622ac444b9aa13ed16b5c6049fb, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/B/399166edb9e34453a3a2f527f9905029, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/B/fdb844eb97b3476999382e9f5e79839c, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/B/93b64512080f46199d5a835131d9a1bf, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/B/d261a845a615453ab8b9676816fdc2c3] to archive 2023-04-25 02:54:35,153 DEBUG [StoreCloser-TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-25 02:54:35,154 DEBUG [StoreCloser-TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/B/9d672d5b816b4f128fd34506fb1fa281 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/B/9d672d5b816b4f128fd34506fb1fa281 2023-04-25 02:54:35,155 DEBUG [StoreCloser-TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/B/e411495647fd4bfe994149ae41b61d23 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/B/e411495647fd4bfe994149ae41b61d23 2023-04-25 02:54:35,156 DEBUG [StoreCloser-TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/B/2500b86ed0524a3488809f7eb6811311 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/B/2500b86ed0524a3488809f7eb6811311 2023-04-25 02:54:35,157 DEBUG [StoreCloser-TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/B/ad7625f91b8645ad838990f1acc5878e to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/B/ad7625f91b8645ad838990f1acc5878e 2023-04-25 02:54:35,158 DEBUG [StoreCloser-TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/B/e2f4d75feec94c57b6eb0c32ef4fb5fa to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/B/e2f4d75feec94c57b6eb0c32ef4fb5fa 2023-04-25 02:54:35,159 DEBUG [StoreCloser-TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/B/8b574ddc194a4d29b92fcf273e5ce506 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/B/8b574ddc194a4d29b92fcf273e5ce506 2023-04-25 02:54:35,160 DEBUG [StoreCloser-TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/B/67f93a6b068a4cde974592dae873f4b3 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/B/67f93a6b068a4cde974592dae873f4b3 2023-04-25 02:54:35,161 DEBUG [StoreCloser-TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/B/bcce52a27cb64fca8262217a8ec2c06e to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/B/bcce52a27cb64fca8262217a8ec2c06e 2023-04-25 02:54:35,162 DEBUG [StoreCloser-TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/B/c40efdd9c0884675ba292c302497098f to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/B/c40efdd9c0884675ba292c302497098f 2023-04-25 02:54:35,163 DEBUG [StoreCloser-TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/B/522cb9c8f84b426181ee9bb7dc9fc27e to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/B/522cb9c8f84b426181ee9bb7dc9fc27e 2023-04-25 02:54:35,164 DEBUG [StoreCloser-TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/B/314f2f0378cb41f58618a457bc84285f to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/B/314f2f0378cb41f58618a457bc84285f 2023-04-25 02:54:35,165 DEBUG [StoreCloser-TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/B/0ab45ceb5c454b62b269dd7d54345b64 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/B/0ab45ceb5c454b62b269dd7d54345b64 2023-04-25 02:54:35,166 DEBUG [StoreCloser-TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/B/4c8aee74a42f4267aae1cfc5c4096dc6 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/B/4c8aee74a42f4267aae1cfc5c4096dc6 2023-04-25 02:54:35,166 DEBUG [StoreCloser-TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/B/55550be5974a4c6c82dc75370e638ac3 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/B/55550be5974a4c6c82dc75370e638ac3 2023-04-25 02:54:35,167 DEBUG [StoreCloser-TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/B/107b8acc22fb42ec84647cfd2033d076 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/B/107b8acc22fb42ec84647cfd2033d076 2023-04-25 02:54:35,169 DEBUG [StoreCloser-TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/B/5e8755f5eada409bb76a1b107a3795a3 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/B/5e8755f5eada409bb76a1b107a3795a3 2023-04-25 02:54:35,170 DEBUG [StoreCloser-TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/B/4438dd49f6794adf822d1deab57fa16e to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/B/4438dd49f6794adf822d1deab57fa16e 2023-04-25 02:54:35,171 DEBUG [StoreCloser-TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/B/0f07c3654503421995bc37dea0c599c9 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/B/0f07c3654503421995bc37dea0c599c9 2023-04-25 02:54:35,173 DEBUG [StoreCloser-TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/B/a72ad622ac444b9aa13ed16b5c6049fb to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/B/a72ad622ac444b9aa13ed16b5c6049fb 2023-04-25 02:54:35,174 DEBUG [StoreCloser-TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/B/399166edb9e34453a3a2f527f9905029 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/B/399166edb9e34453a3a2f527f9905029 2023-04-25 02:54:35,175 DEBUG [StoreCloser-TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/B/fdb844eb97b3476999382e9f5e79839c to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/B/fdb844eb97b3476999382e9f5e79839c 2023-04-25 02:54:35,175 DEBUG [StoreCloser-TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/B/93b64512080f46199d5a835131d9a1bf to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/B/93b64512080f46199d5a835131d9a1bf 2023-04-25 02:54:35,177 DEBUG [StoreCloser-TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/B/d261a845a615453ab8b9676816fdc2c3 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/B/d261a845a615453ab8b9676816fdc2c3 2023-04-25 02:54:35,180 DEBUG [StoreCloser-TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/C/ea45787746b740a88796fac1b681bb91, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/C/d0dcc153e64a47cd9669afca16d2538f, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/C/d4d62a806d6a48a380c0e88b0525fca1, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/C/3106d5b1f8134a99ad4d07af64836bd8, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/C/6a48af4e5ef1489fb6a59a5bd89957b5, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/C/a961f1057d05493a80e87dc1f76b1fdc, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/C/9b1b72180ef444c7aca5c54bc9105341, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/C/182dc9262d114df490a115aefff1c753, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/C/b929f736a8b24d74be77fe058d07cf43, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/C/6e9b68a842b74489b4d23700a95e60dd, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/C/17375cdc295943b5a308615ca8fe5afc, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/C/14590354535f4d76854ba539310fa1ec, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/C/a6645645b71c40658dd3f3c2f72ad5ff, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/C/e753733aec9a45de8dd3e15bead75896, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/C/20d346642b6a4da487b41cdbd8dff658, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/C/d21a7e227b4d48eab8f1543506ee34bb, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/C/66fc279401bf46148fb40cb78196fca4, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/C/0792d4b169b54c9a8b0bfed88ed368a4, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/C/6a2fcabbc73a4653822826c08ef3d003, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/C/4493fcfe015045cc9e6fb5be697e4222, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/C/4bd8e8ee898548f8bad0ceaa0083ac8d, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/C/f69ffe4f9d124c66be13da18e5ad11e4, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/C/4c0bb33453134b19a348bf4734aa994f] to archive 2023-04-25 02:54:35,180 DEBUG [StoreCloser-TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-25 02:54:35,182 DEBUG [StoreCloser-TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/C/ea45787746b740a88796fac1b681bb91 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/C/ea45787746b740a88796fac1b681bb91 2023-04-25 02:54:35,183 DEBUG [StoreCloser-TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/C/d0dcc153e64a47cd9669afca16d2538f to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/C/d0dcc153e64a47cd9669afca16d2538f 2023-04-25 02:54:35,184 DEBUG [StoreCloser-TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/C/d4d62a806d6a48a380c0e88b0525fca1 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/C/d4d62a806d6a48a380c0e88b0525fca1 2023-04-25 02:54:35,185 DEBUG [StoreCloser-TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/C/3106d5b1f8134a99ad4d07af64836bd8 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/C/3106d5b1f8134a99ad4d07af64836bd8 2023-04-25 02:54:35,186 DEBUG [StoreCloser-TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/C/6a48af4e5ef1489fb6a59a5bd89957b5 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/C/6a48af4e5ef1489fb6a59a5bd89957b5 2023-04-25 02:54:35,187 DEBUG [StoreCloser-TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/C/a961f1057d05493a80e87dc1f76b1fdc to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/C/a961f1057d05493a80e87dc1f76b1fdc 2023-04-25 02:54:35,188 DEBUG [StoreCloser-TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/C/9b1b72180ef444c7aca5c54bc9105341 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/C/9b1b72180ef444c7aca5c54bc9105341 2023-04-25 02:54:35,189 DEBUG [StoreCloser-TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/C/182dc9262d114df490a115aefff1c753 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/C/182dc9262d114df490a115aefff1c753 2023-04-25 02:54:35,190 DEBUG [StoreCloser-TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/C/b929f736a8b24d74be77fe058d07cf43 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/C/b929f736a8b24d74be77fe058d07cf43 2023-04-25 02:54:35,191 DEBUG [StoreCloser-TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/C/6e9b68a842b74489b4d23700a95e60dd to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/C/6e9b68a842b74489b4d23700a95e60dd 2023-04-25 02:54:35,192 DEBUG [StoreCloser-TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/C/17375cdc295943b5a308615ca8fe5afc to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/C/17375cdc295943b5a308615ca8fe5afc 2023-04-25 02:54:35,193 DEBUG [StoreCloser-TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/C/14590354535f4d76854ba539310fa1ec to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/C/14590354535f4d76854ba539310fa1ec 2023-04-25 02:54:35,194 DEBUG [StoreCloser-TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/C/a6645645b71c40658dd3f3c2f72ad5ff to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/C/a6645645b71c40658dd3f3c2f72ad5ff 2023-04-25 02:54:35,195 DEBUG [StoreCloser-TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/C/e753733aec9a45de8dd3e15bead75896 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/C/e753733aec9a45de8dd3e15bead75896 2023-04-25 02:54:35,196 DEBUG [StoreCloser-TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/C/20d346642b6a4da487b41cdbd8dff658 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/C/20d346642b6a4da487b41cdbd8dff658 2023-04-25 02:54:35,197 DEBUG [StoreCloser-TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/C/d21a7e227b4d48eab8f1543506ee34bb to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/C/d21a7e227b4d48eab8f1543506ee34bb 2023-04-25 02:54:35,198 DEBUG [StoreCloser-TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/C/66fc279401bf46148fb40cb78196fca4 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/C/66fc279401bf46148fb40cb78196fca4 2023-04-25 02:54:35,199 DEBUG [StoreCloser-TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/C/0792d4b169b54c9a8b0bfed88ed368a4 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/C/0792d4b169b54c9a8b0bfed88ed368a4 2023-04-25 02:54:35,200 DEBUG [StoreCloser-TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/C/6a2fcabbc73a4653822826c08ef3d003 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/C/6a2fcabbc73a4653822826c08ef3d003 2023-04-25 02:54:35,201 DEBUG [StoreCloser-TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/C/4493fcfe015045cc9e6fb5be697e4222 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/C/4493fcfe015045cc9e6fb5be697e4222 2023-04-25 02:54:35,202 DEBUG [StoreCloser-TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/C/4bd8e8ee898548f8bad0ceaa0083ac8d to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/C/4bd8e8ee898548f8bad0ceaa0083ac8d 2023-04-25 02:54:35,203 DEBUG [StoreCloser-TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/C/f69ffe4f9d124c66be13da18e5ad11e4 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/C/f69ffe4f9d124c66be13da18e5ad11e4 2023-04-25 02:54:35,204 DEBUG [StoreCloser-TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/C/4c0bb33453134b19a348bf4734aa994f to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/C/4c0bb33453134b19a348bf4734aa994f 2023-04-25 02:54:35,208 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/recovered.edits/473.seqid, newMaxSeqId=473, maxSeqId=4 2023-04-25 02:54:35,210 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198. 2023-04-25 02:54:35,210 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 388bc1373cd34a10ba8be8ddc731e198: 2023-04-25 02:54:35,211 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 388bc1373cd34a10ba8be8ddc731e198 2023-04-25 02:54:35,212 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=25 updating hbase:meta row=388bc1373cd34a10ba8be8ddc731e198, regionState=CLOSED 2023-04-25 02:54:35,212 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682391275211"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682391275211"}]},"ts":"1682391275211"} 2023-04-25 02:54:35,214 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=26, resume processing ppid=25 2023-04-25 02:54:35,214 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=26, ppid=25, state=SUCCESS; CloseRegionProcedure 388bc1373cd34a10ba8be8ddc731e198, server=jenkins-hbase4.apache.org,46697,1682391220023 in 1.9180 sec 2023-04-25 02:54:35,215 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=25, resume processing ppid=24 2023-04-25 02:54:35,215 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=25, ppid=24, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=388bc1373cd34a10ba8be8ddc731e198, UNASSIGN in 1.9220 sec 2023-04-25 02:54:35,216 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682391275216"}]},"ts":"1682391275216"} 2023-04-25 02:54:35,217 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-25 02:54:35,218 INFO [PEWorker-1] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-25 02:54:35,220 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=24, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.9350 sec 2023-04-25 02:54:35,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-25 02:54:35,392 INFO [Listener at localhost/41915] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 24 completed 2023-04-25 02:54:35,393 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-04-25 02:54:35,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure2.ProcedureExecutor(1029): Stored pid=27, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-25 02:54:35,395 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=27, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-25 02:54:35,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1227): Checking to see if procedure is done pid=27 2023-04-25 02:54:35,396 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=27, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-25 02:54:35,401 DEBUG [HFileArchiver-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/.tmp/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198 2023-04-25 02:54:35,403 DEBUG [HFileArchiver-5] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/.tmp/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/A, FileablePath, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/.tmp/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/B, FileablePath, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/.tmp/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/C, FileablePath, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/.tmp/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/recovered.edits] 2023-04-25 02:54:35,406 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/.tmp/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/A/37060278023d46adae9b2384f3071028 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/A/37060278023d46adae9b2384f3071028 2023-04-25 02:54:35,407 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/.tmp/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/A/51a2226ab5e944d9847136275506a7dd to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/A/51a2226ab5e944d9847136275506a7dd 2023-04-25 02:54:35,409 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/.tmp/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/A/70652f454fa54e06a6943c7dc7ddaa3c to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/A/70652f454fa54e06a6943c7dc7ddaa3c 2023-04-25 02:54:35,410 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/.tmp/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/A/98ed6608a8684edba57ea59d209f0bfd to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/A/98ed6608a8684edba57ea59d209f0bfd 2023-04-25 02:54:35,411 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/.tmp/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/A/d0e06dbb28f64ccb9494d2dd74264e4b to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/A/d0e06dbb28f64ccb9494d2dd74264e4b 2023-04-25 02:54:35,413 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/.tmp/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/B/22be35f2887e4d859f2cc2785dd1a1fd to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/B/22be35f2887e4d859f2cc2785dd1a1fd 2023-04-25 02:54:35,414 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/.tmp/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/B/856ada7d982948a0a9d62cbd92880571 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/B/856ada7d982948a0a9d62cbd92880571 2023-04-25 02:54:35,416 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/.tmp/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/B/ae9982eb13a84fd4adfd2687f6745d33 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/B/ae9982eb13a84fd4adfd2687f6745d33 2023-04-25 02:54:35,417 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/.tmp/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/B/eb3e71697d8a4fea92a7fcd8d60f15f6 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/B/eb3e71697d8a4fea92a7fcd8d60f15f6 2023-04-25 02:54:35,418 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/.tmp/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/B/f61be2cd26f549a69ce1eb9593f8445a to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/B/f61be2cd26f549a69ce1eb9593f8445a 2023-04-25 02:54:35,420 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/.tmp/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/C/05b5e3d7e1064f4584a13593b6714c29 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/C/05b5e3d7e1064f4584a13593b6714c29 2023-04-25 02:54:35,421 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/.tmp/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/C/5ebf3082bbe944acb544ab7ad05afb93 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/C/5ebf3082bbe944acb544ab7ad05afb93 2023-04-25 02:54:35,423 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/.tmp/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/C/8385df4aab19453d94a0f06461e796a1 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/C/8385df4aab19453d94a0f06461e796a1 2023-04-25 02:54:35,424 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/.tmp/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/C/9f1df4a42e1d462c8d63fcbc975388a5 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/C/9f1df4a42e1d462c8d63fcbc975388a5 2023-04-25 02:54:35,425 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/.tmp/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/C/eed598aa6b89408dabbb06f0bff30e06 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/C/eed598aa6b89408dabbb06f0bff30e06 2023-04-25 02:54:35,428 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/.tmp/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/recovered.edits/473.seqid to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198/recovered.edits/473.seqid 2023-04-25 02:54:35,428 DEBUG [HFileArchiver-5] backup.HFileArchiver(596): Deleted hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/.tmp/data/default/TestAcidGuarantees/388bc1373cd34a10ba8be8ddc731e198 2023-04-25 02:54:35,429 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-25 02:54:35,429 DEBUG [PEWorker-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-25 02:54:35,430 DEBUG [PEWorker-5] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A] 2023-04-25 02:54:35,434 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042509234cdf52b440c8a1178dc2e3476b75 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042509234cdf52b440c8a1178dc2e3476b75 2023-04-25 02:54:35,435 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304251d7fe7daff0042cfbdfe883e02aad585 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304251d7fe7daff0042cfbdfe883e02aad585 2023-04-25 02:54:35,436 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042522672a3f84f845868d98a863ca671072 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042522672a3f84f845868d98a863ca671072 2023-04-25 02:54:35,438 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304252d4f1c7ee46447a9965531e6515d5c31 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304252d4f1c7ee46447a9965531e6515d5c31 2023-04-25 02:54:35,439 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304252ea2ea1a45194e7db499e2226b1d1efa to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304252ea2ea1a45194e7db499e2226b1d1efa 2023-04-25 02:54:35,441 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425472c1c0fc9af423f9ad49363b35bbb5d to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425472c1c0fc9af423f9ad49363b35bbb5d 2023-04-25 02:54:35,442 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042556232c6992ab4bfdaab910ea58c94ba0 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042556232c6992ab4bfdaab910ea58c94ba0 2023-04-25 02:54:35,443 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304256e710fb3b03746518ae343e932926944 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304256e710fb3b03746518ae343e932926944 2023-04-25 02:54:35,444 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304257f76aa0f7efd4409861200c8d48b9805 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304257f76aa0f7efd4409861200c8d48b9805 2023-04-25 02:54:35,445 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304258cab597f5b844a98acc69582b575ac88 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304258cab597f5b844a98acc69582b575ac88 2023-04-25 02:54:35,446 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042595e83d6c0fb349a1976f5ca5e5c75814 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042595e83d6c0fb349a1976f5ca5e5c75814 2023-04-25 02:54:35,447 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304259eaaeea5f8cd493f891fd5706af01f0a to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304259eaaeea5f8cd493f891fd5706af01f0a 2023-04-25 02:54:35,449 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425a6a8b50d295b4aebb142f3803176ed25 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425a6a8b50d295b4aebb142f3803176ed25 2023-04-25 02:54:35,450 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425a7597d1dff174e40acab2a42c6710902 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425a7597d1dff174e40acab2a42c6710902 2023-04-25 02:54:35,451 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425b25966fba6f24010a6f988c9625d55f4 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425b25966fba6f24010a6f988c9625d55f4 2023-04-25 02:54:35,452 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425b5687f1285094b4aa00f8c27b639a8d5 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425b5687f1285094b4aa00f8c27b639a8d5 2023-04-25 02:54:35,453 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425c0c17081c93644cc98a53e99e390dd50 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425c0c17081c93644cc98a53e99e390dd50 2023-04-25 02:54:35,454 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425c2cda85169824b2cbc04bd084b8d8d84 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425c2cda85169824b2cbc04bd084b8d8d84 2023-04-25 02:54:35,455 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425e381582adbf148c3af83695597a589f2 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425e381582adbf148c3af83695597a589f2 2023-04-25 02:54:35,456 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425e4bfb13b205540529c65499ae8911d24 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425e4bfb13b205540529c65499ae8911d24 2023-04-25 02:54:35,457 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425f3fff75d996540109428076c602f32ed to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425f3fff75d996540109428076c602f32ed 2023-04-25 02:54:35,457 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425f5162868956248fc86720a6a083b4b04 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425f5162868956248fc86720a6a083b4b04 2023-04-25 02:54:35,458 DEBUG [PEWorker-5] backup.HFileArchiver(596): Deleted hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-25 02:54:35,460 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=27, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-25 02:54:35,462 WARN [PEWorker-5] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-25 02:54:35,464 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-25 02:54:35,465 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=27, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-25 02:54:35,465 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-25 02:54:35,465 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1682391275465"}]},"ts":"9223372036854775807"} 2023-04-25 02:54:35,466 INFO [PEWorker-5] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-25 02:54:35,466 DEBUG [PEWorker-5] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 388bc1373cd34a10ba8be8ddc731e198, NAME => 'TestAcidGuarantees,,1682391248837.388bc1373cd34a10ba8be8ddc731e198.', STARTKEY => '', ENDKEY => ''}] 2023-04-25 02:54:35,466 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-25 02:54:35,466 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1682391275466"}]},"ts":"9223372036854775807"} 2023-04-25 02:54:35,468 INFO [PEWorker-5] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-25 02:54:35,469 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(130): Finished pid=27, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-25 02:54:35,470 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=27, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 76 msec 2023-04-25 02:54:35,497 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1227): Checking to see if procedure is done pid=27 2023-04-25 02:54:35,497 INFO [Listener at localhost/41915] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 27 completed 2023-04-25 02:54:35,510 INFO [Listener at localhost/41915] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMobMixedAtomicity Thread=238 (was 235) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-599764414_15 at /127.0.0.1:51472 [Waiting for operation #3] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x14e0c1f2-shared-pool-31 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-15 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-16 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-4 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-599764414_15 at /127.0.0.1:52566 [Waiting for operation #3] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x14e0c1f2-shared-pool-34 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-14 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x14e0c1f2-shared-pool-32 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/68daf6bb-f478-d813-e37b-dbe23bee4a08/cluster_35ab5f43-d241-074f-505d-0f2e954a1696/dfs/data/data2/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/68daf6bb-f478-d813-e37b-dbe23bee4a08/cluster_35ab5f43-d241-074f-505d-0f2e954a1696/dfs/data/data1/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x14e0c1f2-shared-pool-33 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-5 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-599764414_15 at /127.0.0.1:52628 [Waiting for operation #3] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) - Thread LEAK? -, OpenFileDescriptor=559 (was 577), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=500 (was 560), ProcessCount=170 (was 170), AvailableMemoryMB=4306 (was 4485) 2023-04-25 02:54:35,521 INFO [Listener at localhost/41915] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testGetAtomicity Thread=238, OpenFileDescriptor=559, MaxFileDescriptor=60000, SystemLoadAverage=500, ProcessCount=170, AvailableMemoryMB=4305 2023-04-25 02:54:35,523 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-25 02:54:35,523 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-25 02:54:35,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure2.ProcedureExecutor(1029): Stored pid=28, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-25 02:54:35,525 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-25 02:54:35,525 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 28 2023-04-25 02:54:35,526 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-25 02:54:35,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-25 02:54:35,527 DEBUG [HFileArchiver-6] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/.tmp/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b 2023-04-25 02:54:35,527 DEBUG [HFileArchiver-6] backup.HFileArchiver(153): Directory hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/.tmp/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b empty. 2023-04-25 02:54:35,528 DEBUG [HFileArchiver-6] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/.tmp/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b 2023-04-25 02:54:35,528 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-25 02:54:35,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-25 02:54:35,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-25 02:54:35,935 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-25 02:54:35,937 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 40cf532a35b33bde962f4f28e36e887b, NAME => 'TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/.tmp 2023-04-25 02:54:36,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-25 02:54:36,343 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-25 02:54:36,343 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 40cf532a35b33bde962f4f28e36e887b, disabling compactions & flushes 2023-04-25 02:54:36,343 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. 2023-04-25 02:54:36,343 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. 2023-04-25 02:54:36,343 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. after waiting 0 ms 2023-04-25 02:54:36,343 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. 2023-04-25 02:54:36,343 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. 2023-04-25 02:54:36,343 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 40cf532a35b33bde962f4f28e36e887b: 2023-04-25 02:54:36,346 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-25 02:54:36,347 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682391276346"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682391276346"}]},"ts":"1682391276346"} 2023-04-25 02:54:36,348 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-25 02:54:36,348 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-25 02:54:36,348 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682391276348"}]},"ts":"1682391276348"} 2023-04-25 02:54:36,349 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-25 02:54:36,353 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=40cf532a35b33bde962f4f28e36e887b, ASSIGN}] 2023-04-25 02:54:36,355 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=40cf532a35b33bde962f4f28e36e887b, ASSIGN 2023-04-25 02:54:36,355 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=40cf532a35b33bde962f4f28e36e887b, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,46697,1682391220023; forceNewPlan=false, retain=false 2023-04-25 02:54:36,506 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=29 updating hbase:meta row=40cf532a35b33bde962f4f28e36e887b, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:36,507 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682391276506"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682391276506"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682391276506"}]},"ts":"1682391276506"} 2023-04-25 02:54:36,508 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=30, ppid=29, state=RUNNABLE; OpenRegionProcedure 40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023}] 2023-04-25 02:54:36,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-25 02:54:36,663 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. 2023-04-25 02:54:36,663 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 40cf532a35b33bde962f4f28e36e887b, NAME => 'TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b.', STARTKEY => '', ENDKEY => ''} 2023-04-25 02:54:36,663 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 40cf532a35b33bde962f4f28e36e887b 2023-04-25 02:54:36,663 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-25 02:54:36,663 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 40cf532a35b33bde962f4f28e36e887b 2023-04-25 02:54:36,663 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 40cf532a35b33bde962f4f28e36e887b 2023-04-25 02:54:36,664 INFO [StoreOpener-40cf532a35b33bde962f4f28e36e887b-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 40cf532a35b33bde962f4f28e36e887b 2023-04-25 02:54:36,666 DEBUG [StoreOpener-40cf532a35b33bde962f4f28e36e887b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A 2023-04-25 02:54:36,666 DEBUG [StoreOpener-40cf532a35b33bde962f4f28e36e887b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A 2023-04-25 02:54:36,666 INFO [StoreOpener-40cf532a35b33bde962f4f28e36e887b-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-25 02:54:36,666 INFO [StoreOpener-40cf532a35b33bde962f4f28e36e887b-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 40cf532a35b33bde962f4f28e36e887b columnFamilyName A 2023-04-25 02:54:36,667 INFO [StoreOpener-40cf532a35b33bde962f4f28e36e887b-1] regionserver.HStore(310): Store=40cf532a35b33bde962f4f28e36e887b/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-25 02:54:36,667 INFO [StoreOpener-40cf532a35b33bde962f4f28e36e887b-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 40cf532a35b33bde962f4f28e36e887b 2023-04-25 02:54:36,668 DEBUG [StoreOpener-40cf532a35b33bde962f4f28e36e887b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B 2023-04-25 02:54:36,668 DEBUG [StoreOpener-40cf532a35b33bde962f4f28e36e887b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B 2023-04-25 02:54:36,668 INFO [StoreOpener-40cf532a35b33bde962f4f28e36e887b-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-25 02:54:36,668 INFO [StoreOpener-40cf532a35b33bde962f4f28e36e887b-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 40cf532a35b33bde962f4f28e36e887b columnFamilyName B 2023-04-25 02:54:36,669 INFO [StoreOpener-40cf532a35b33bde962f4f28e36e887b-1] regionserver.HStore(310): Store=40cf532a35b33bde962f4f28e36e887b/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-25 02:54:36,669 INFO [StoreOpener-40cf532a35b33bde962f4f28e36e887b-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 40cf532a35b33bde962f4f28e36e887b 2023-04-25 02:54:36,670 DEBUG [StoreOpener-40cf532a35b33bde962f4f28e36e887b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C 2023-04-25 02:54:36,670 DEBUG [StoreOpener-40cf532a35b33bde962f4f28e36e887b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C 2023-04-25 02:54:36,670 INFO [StoreOpener-40cf532a35b33bde962f4f28e36e887b-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-25 02:54:36,670 INFO [StoreOpener-40cf532a35b33bde962f4f28e36e887b-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 40cf532a35b33bde962f4f28e36e887b columnFamilyName C 2023-04-25 02:54:36,671 INFO [StoreOpener-40cf532a35b33bde962f4f28e36e887b-1] regionserver.HStore(310): Store=40cf532a35b33bde962f4f28e36e887b/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-25 02:54:36,671 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. 2023-04-25 02:54:36,671 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b 2023-04-25 02:54:36,672 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b 2023-04-25 02:54:36,674 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-25 02:54:36,675 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 40cf532a35b33bde962f4f28e36e887b 2023-04-25 02:54:36,677 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-25 02:54:36,677 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 40cf532a35b33bde962f4f28e36e887b; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=66502277, jitterRate=-0.009038850665092468}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-25 02:54:36,677 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 40cf532a35b33bde962f4f28e36e887b: 2023-04-25 02:54:36,678 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b., pid=30, masterSystemTime=1682391276659 2023-04-25 02:54:36,679 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. 2023-04-25 02:54:36,679 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. 2023-04-25 02:54:36,679 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=29 updating hbase:meta row=40cf532a35b33bde962f4f28e36e887b, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:36,680 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682391276679"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1682391276679"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1682391276679"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1682391276679"}]},"ts":"1682391276679"} 2023-04-25 02:54:36,682 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=30, resume processing ppid=29 2023-04-25 02:54:36,682 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=30, ppid=29, state=SUCCESS; OpenRegionProcedure 40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 in 173 msec 2023-04-25 02:54:36,684 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=29, resume processing ppid=28 2023-04-25 02:54:36,684 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=29, ppid=28, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=40cf532a35b33bde962f4f28e36e887b, ASSIGN in 329 msec 2023-04-25 02:54:36,684 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-25 02:54:36,684 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682391276684"}]},"ts":"1682391276684"} 2023-04-25 02:54:36,685 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-25 02:54:36,687 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-25 02:54:36,689 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=28, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1640 sec 2023-04-25 02:54:37,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-25 02:54:37,631 INFO [Listener at localhost/41915] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 28 completed 2023-04-25 02:54:37,631 DEBUG [Listener at localhost/41915] zookeeper.ReadOnlyZKClient(139): Connect 0x5c124820 to 127.0.0.1:62054 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-25 02:54:37,636 DEBUG [Listener at localhost/41915] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@34b7fed0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-25 02:54:37,637 DEBUG [Listener at localhost/41915] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-25 02:54:37,638 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59138, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-25 02:54:37,639 DEBUG [Listener at localhost/41915] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-25 02:54:37,641 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:48012, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-25 02:54:37,642 DEBUG [Listener at localhost/41915] zookeeper.ReadOnlyZKClient(139): Connect 0x3157d4df to 127.0.0.1:62054 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-25 02:54:37,647 DEBUG [Listener at localhost/41915] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7f4ad99d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-25 02:54:37,647 DEBUG [Listener at localhost/41915] zookeeper.ReadOnlyZKClient(139): Connect 0x3cf5bdb7 to 127.0.0.1:62054 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-25 02:54:37,651 DEBUG [Listener at localhost/41915] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@330b77da, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-25 02:54:37,651 DEBUG [Listener at localhost/41915] zookeeper.ReadOnlyZKClient(139): Connect 0x6919b2a0 to 127.0.0.1:62054 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-25 02:54:37,654 DEBUG [Listener at localhost/41915] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7810a231, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-25 02:54:37,654 DEBUG [Listener at localhost/41915] zookeeper.ReadOnlyZKClient(139): Connect 0x2ce5573e to 127.0.0.1:62054 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-25 02:54:37,658 DEBUG [Listener at localhost/41915] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@8cd7cb0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-25 02:54:37,658 DEBUG [Listener at localhost/41915] zookeeper.ReadOnlyZKClient(139): Connect 0x087b3095 to 127.0.0.1:62054 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-25 02:54:37,662 DEBUG [Listener at localhost/41915] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@ea1ca03, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-25 02:54:37,662 DEBUG [Listener at localhost/41915] zookeeper.ReadOnlyZKClient(139): Connect 0x433f251d to 127.0.0.1:62054 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-25 02:54:37,666 DEBUG [Listener at localhost/41915] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@434e0eab, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-25 02:54:37,666 DEBUG [Listener at localhost/41915] zookeeper.ReadOnlyZKClient(139): Connect 0x66ee0bd0 to 127.0.0.1:62054 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-25 02:54:37,670 DEBUG [Listener at localhost/41915] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5c43dddd, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-25 02:54:37,670 DEBUG [Listener at localhost/41915] zookeeper.ReadOnlyZKClient(139): Connect 0x3b4658f8 to 127.0.0.1:62054 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-25 02:54:37,673 DEBUG [Listener at localhost/41915] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2de064cb, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-25 02:54:37,673 DEBUG [Listener at localhost/41915] zookeeper.ReadOnlyZKClient(139): Connect 0x5d2b992d to 127.0.0.1:62054 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-25 02:54:37,679 DEBUG [Listener at localhost/41915] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@54a39be8, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-25 02:54:37,679 DEBUG [Listener at localhost/41915] zookeeper.ReadOnlyZKClient(139): Connect 0x6a98265c to 127.0.0.1:62054 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-25 02:54:37,682 DEBUG [Listener at localhost/41915] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6bfb38d3, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-25 02:54:37,687 DEBUG [hconnection-0x2b59d60a-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-25 02:54:37,687 DEBUG [hconnection-0x3cbfcdac-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-25 02:54:37,687 DEBUG [hconnection-0x3599612a-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-25 02:54:37,687 DEBUG [hconnection-0x62f6a859-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-25 02:54:37,688 DEBUG [hconnection-0x684bf0bf-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-25 02:54:37,689 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59152, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-25 02:54:37,687 DEBUG [hconnection-0x774150a2-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-25 02:54:37,687 DEBUG [hconnection-0x297cf8eb-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-25 02:54:37,689 DEBUG [hconnection-0xb8e8661-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-25 02:54:37,689 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59160, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-25 02:54:37,690 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59158, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-25 02:54:37,691 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:54:37,691 DEBUG [hconnection-0x42b68ec2-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-25 02:54:37,692 DEBUG [hconnection-0x29789a8a-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-25 02:54:37,695 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59182, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-25 02:54:37,695 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59214, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-25 02:54:37,698 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59180, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-25 02:54:37,698 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59198, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-25 02:54:37,701 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59178, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-25 02:54:37,701 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59186, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-25 02:54:37,702 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59216, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-25 02:54:37,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:54:37,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:54:37,704 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:54:37,704 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:37,704 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:54:37,704 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:54:37,705 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:37,705 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:37,708 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:37,708 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:37,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:37,708 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:37,710 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:37,710 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:54:37,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:37,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(9158): Flush requested on 40cf532a35b33bde962f4f28e36e887b 2023-04-25 02:54:37,711 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 40cf532a35b33bde962f4f28e36e887b 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-25 02:54:37,711 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 40cf532a35b33bde962f4f28e36e887b, store=A 2023-04-25 02:54:37,711 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:54:37,711 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 40cf532a35b33bde962f4f28e36e887b, store=B 2023-04-25 02:54:37,711 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:54:37,712 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 40cf532a35b33bde962f4f28e36e887b, store=C 2023-04-25 02:54:37,712 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:54:37,712 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:37,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:54:37,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:37,712 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:54:37,713 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:54:37,713 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:54:37,713 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:54:37,713 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:37,713 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:54:37,713 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:54:37,713 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:54:37,713 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. 2023-04-25 02:54:37,714 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. started... 2023-04-25 02:54:37,714 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. as already flushing 2023-04-25 02:54:37,714 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. 2023-04-25 02:54:37,714 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:54:37,714 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:54:37,714 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:37,714 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:54:37,714 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:54:37,715 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:37,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:37,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:37,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:37,715 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:37,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:37,716 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:37,716 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:54:37,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:37,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:37,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:37,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:37,717 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:54:37,717 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@605d346e[Count = 0] remaining members to acquire global barrier 2023-04-25 02:54:37,717 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:54:37,718 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:37,720 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:37,720 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:37,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:37,720 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:54:37,720 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:54:37,720 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:37,720 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:54:37,720 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:54:37,721 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:37,721 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:54:37,722 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:37,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:37,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:37,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:37,722 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:54:37,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:37,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:37,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:37,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:37,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:37,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:37,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:37,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:54:37,725 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:54:37,725 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:54:37,725 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:54:37,725 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:54:37,726 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:54:37,726 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:54:37,728 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:37,728 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:37,728 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:37,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:37,728 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:37,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:37,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:37,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:37,728 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:37,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:37,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:37,729 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:37,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:37,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:37,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:37,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:37,730 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:37,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:37,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:37,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:37,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:37,732 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:37,732 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:37,732 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:37,732 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:37,732 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:54:37,732 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:54:37,732 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:37,732 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:37,732 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:37,732 DEBUG [Thread-1512] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:54:37,732 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:37,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:37,733 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:54:37,732 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:37,733 DEBUG [Thread-1512] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:54:37,733 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:37,733 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:37,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:37,733 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:37,735 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682391277725, max timestamp=1682391277734Num uniques 859; 2023-04-25 02:54:37,735 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-25 02:54:37,736 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682391277725, max timestamp=1682391277734Num uniques 859; 2023-04-25 02:54:37,736 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-25 02:54:37,737 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682391277725, max timestamp=1682391277734Num uniques 859; 2023-04-25 02:54:37,737 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-25 02:54:37,745 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=12 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp/A/92c9b725031443e5b38491daade551ee 2023-04-25 02:54:37,752 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:37,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59214 deadline: 1682391337748, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:37,753 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:37,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59186 deadline: 1682391337749, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:37,753 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:37,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59216 deadline: 1682391337750, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:37,755 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:37,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59158 deadline: 1682391337752, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:37,755 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:37,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59160 deadline: 1682391337752, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:37,791 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=12 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp/B/370a415aaab24a6da9e885fe53ad6176 2023-04-25 02:54:37,833 DEBUG [Thread-1512] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:54:37,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:54:37,836 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:54:37,857 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:37,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59214 deadline: 1682391337853, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:37,857 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:37,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59186 deadline: 1682391337854, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:37,857 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:37,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59216 deadline: 1682391337854, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:37,858 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:37,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59160 deadline: 1682391337857, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:37,859 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:37,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59158 deadline: 1682391337857, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:37,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:54:37,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:54:37,877 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:54:37,877 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:37,877 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:54:37,877 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:54:37,878 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:37,878 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:37,880 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:37,880 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:37,880 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:37,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:37,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:37,881 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:37,881 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:54:37,881 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:37,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:54:37,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:37,881 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:54:37,896 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:54:37,896 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:54:37,896 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:54:37,896 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:37,896 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:54:37,904 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:54:37,904 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:54:37,904 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. 2023-04-25 02:54:37,904 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. started... 2023-04-25 02:54:37,904 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. as already flushing 2023-04-25 02:54:37,904 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. 2023-04-25 02:54:37,904 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:54:37,904 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:54:37,904 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:37,904 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:54:37,904 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:54:37,906 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:37,906 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:37,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:37,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:37,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:37,907 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:37,907 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:54:37,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:37,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:37,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:37,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:37,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:37,908 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:54:37,908 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5485ff04[Count = 0] remaining members to acquire global barrier 2023-04-25 02:54:37,908 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:54:37,909 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:37,911 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:37,911 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:37,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:37,911 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:54:37,911 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:54:37,911 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:54:37,912 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:37,912 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:54:37,918 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:54:37,919 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:37,919 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:54:37,919 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:37,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:37,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:37,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:37,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:37,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:37,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:37,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:37,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:37,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:37,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:37,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:54:37,922 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:54:37,922 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:54:37,922 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:54:37,922 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:54:37,922 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:54:37,922 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:54:37,924 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:37,924 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:37,924 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:37,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:37,924 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:37,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:37,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:37,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:37,924 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:37,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:37,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:37,925 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:37,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:37,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:37,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:37,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:37,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:37,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:37,927 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:37,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:37,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:37,931 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:37,931 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:37,931 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:37,931 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:37,931 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:37,931 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:54:37,931 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:54:37,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:37,931 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:37,931 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:37,932 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:37,932 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:37,932 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:37,932 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:37,932 DEBUG [Thread-1512] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:54:37,932 DEBUG [Thread-1512] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:54:37,932 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:37,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:38,032 DEBUG [Thread-1512] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:54:38,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:54:38,034 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:54:38,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:54:38,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:54:38,035 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:54:38,035 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:38,035 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:54:38,035 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:54:38,036 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:38,036 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:38,038 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:38,038 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:38,038 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:38,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:38,039 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:38,039 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:54:38,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:38,039 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:38,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:54:38,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:38,039 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:54:38,040 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:54:38,040 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:54:38,040 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:54:38,041 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:38,041 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:54:38,041 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:54:38,041 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:54:38,041 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. 2023-04-25 02:54:38,041 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. started... 2023-04-25 02:54:38,041 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. as already flushing 2023-04-25 02:54:38,041 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. 2023-04-25 02:54:38,041 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:54:38,041 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:54:38,041 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:38,041 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:54:38,041 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:54:38,042 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:38,042 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:38,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:38,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:38,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:38,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:38,043 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:38,043 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:54:38,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:38,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:38,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:38,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:38,044 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:54:38,044 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5703293b[Count = 0] remaining members to acquire global barrier 2023-04-25 02:54:38,044 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:54:38,045 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:38,045 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:38,046 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:38,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:38,046 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:54:38,046 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:54:38,046 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:54:38,046 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:38,046 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:54:38,047 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:38,047 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:54:38,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:38,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:38,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:38,047 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:38,047 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:54:38,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:38,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:38,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:38,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:38,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:38,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:38,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:38,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:54:38,049 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:54:38,049 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:54:38,049 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:54:38,049 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:54:38,049 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:54:38,049 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:54:38,052 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:38,052 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:38,052 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:38,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:38,052 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:38,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:38,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:38,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:38,052 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:38,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:38,053 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:38,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:38,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:38,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:38,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:38,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:38,054 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:38,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:38,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:38,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:38,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:38,056 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:38,056 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:38,056 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:38,056 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:38,056 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:38,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:38,056 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:38,056 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:38,056 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:38,056 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:38,056 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:54:38,056 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:54:38,056 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:54:38,056 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:38,056 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:38,056 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:38,057 DEBUG [Thread-1512] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:54:38,057 DEBUG [Thread-1512] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:54:38,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:38,060 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:38,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59186 deadline: 1682391338058, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:38,060 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:38,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59216 deadline: 1682391338059, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:38,062 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:38,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59160 deadline: 1682391338060, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:38,063 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:38,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59158 deadline: 1682391338061, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:38,067 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:38,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59214 deadline: 1682391338065, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:38,157 DEBUG [Thread-1512] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:54:38,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:54:38,159 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:54:38,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:54:38,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:54:38,160 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:54:38,160 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:38,160 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:54:38,160 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:54:38,160 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:38,160 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:38,162 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:38,162 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:38,162 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:38,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:38,162 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:38,162 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:54:38,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:38,162 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:38,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:54:38,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:38,163 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:54:38,163 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:54:38,163 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:54:38,163 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:54:38,163 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:38,163 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:54:38,163 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:54:38,163 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:54:38,163 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. 2023-04-25 02:54:38,163 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. started... 2023-04-25 02:54:38,163 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. as already flushing 2023-04-25 02:54:38,163 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. 2023-04-25 02:54:38,163 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:54:38,164 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:54:38,164 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:38,164 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:54:38,164 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:54:38,165 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:38,165 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:38,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:38,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:38,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:38,165 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:38,165 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:54:38,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:38,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:38,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:38,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:38,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:38,166 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:54:38,166 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2dee6cb9[Count = 0] remaining members to acquire global barrier 2023-04-25 02:54:38,166 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:54:38,167 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:38,169 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:38,169 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:38,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:38,169 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:54:38,169 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:54:38,169 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:54:38,169 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:38,169 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:54:38,170 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:38,170 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:54:38,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:38,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:38,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:38,170 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:38,171 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:54:38,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:38,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:38,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:38,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:38,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:38,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:38,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:38,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:54:38,173 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:54:38,173 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:54:38,173 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:54:38,173 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:54:38,173 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:54:38,173 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:54:38,174 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:38,175 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:38,175 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:38,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:38,175 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:38,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:38,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:38,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:38,175 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:38,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:38,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:38,176 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:38,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:38,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:38,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:38,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:38,177 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:38,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:38,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:38,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:38,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:38,179 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:38,179 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:38,179 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:38,179 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:38,179 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:54:38,179 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:54:38,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:38,179 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:38,179 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:38,179 DEBUG [Thread-1512] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:54:38,179 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:38,179 DEBUG [Thread-1512] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:54:38,179 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:38,179 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:54:38,179 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:38,179 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:38,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:38,179 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:38,180 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:38,276 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=12 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp/C/017f6cf1430b4dc5a4b4e605ead65115 2023-04-25 02:54:38,279 DEBUG [Thread-1512] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:54:38,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:54:38,282 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp/A/92c9b725031443e5b38491daade551ee as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A/92c9b725031443e5b38491daade551ee 2023-04-25 02:54:38,282 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:54:38,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:54:38,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:54:38,284 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:54:38,284 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:38,284 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:54:38,284 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:54:38,284 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:38,284 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:38,286 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A/92c9b725031443e5b38491daade551ee, entries=150, sequenceid=12, filesize=11.6 K 2023-04-25 02:54:38,286 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:38,286 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:38,286 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:38,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:38,286 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:38,286 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:54:38,286 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp/B/370a415aaab24a6da9e885fe53ad6176 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B/370a415aaab24a6da9e885fe53ad6176 2023-04-25 02:54:38,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:38,287 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:38,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:54:38,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:38,287 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:54:38,288 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:54:38,288 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:54:38,288 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:54:38,288 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:38,288 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:54:38,288 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:54:38,288 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:54:38,289 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. 2023-04-25 02:54:38,289 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. started... 2023-04-25 02:54:38,289 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. as already flushing 2023-04-25 02:54:38,289 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. 2023-04-25 02:54:38,289 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:54:38,289 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:54:38,289 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:38,289 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:54:38,289 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:54:38,290 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:38,290 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:38,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:38,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:38,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:38,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:38,291 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:38,291 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:54:38,291 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B/370a415aaab24a6da9e885fe53ad6176, entries=150, sequenceid=12, filesize=11.6 K 2023-04-25 02:54:38,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:38,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:38,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:38,292 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp/C/017f6cf1430b4dc5a4b4e605ead65115 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C/017f6cf1430b4dc5a4b4e605ead65115 2023-04-25 02:54:38,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:38,293 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:54:38,293 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2419d32e[Count = 0] remaining members to acquire global barrier 2023-04-25 02:54:38,293 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:54:38,293 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:38,294 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:38,294 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:38,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:38,294 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:54:38,294 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:54:38,295 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:54:38,295 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:38,295 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:54:38,296 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:54:38,297 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:38,297 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:54:38,297 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:38,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:38,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:38,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:38,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:38,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:38,298 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C/017f6cf1430b4dc5a4b4e605ead65115, entries=150, sequenceid=12, filesize=11.6 K 2023-04-25 02:54:38,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:38,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:38,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:38,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:38,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:38,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:54:38,300 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:54:38,300 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:54:38,300 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:54:38,301 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:54:38,301 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:54:38,301 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:54:38,306 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:38,306 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:38,306 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:38,306 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=136.42 KB/139690 for 40cf532a35b33bde962f4f28e36e887b in 595ms, sequenceid=12, compaction requested=false 2023-04-25 02:54:38,306 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:38,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:38,306 DEBUG [MemStoreFlusher.0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'TestAcidGuarantees' 2023-04-25 02:54:38,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:38,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:38,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:38,306 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:38,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:38,307 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:38,307 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 40cf532a35b33bde962f4f28e36e887b: 2023-04-25 02:54:38,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:38,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:38,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:38,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:38,309 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:38,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:38,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:38,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:38,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:38,316 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:38,316 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:38,316 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:38,316 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:38,316 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:54:38,316 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:54:38,316 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:38,316 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:38,316 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:38,316 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:38,316 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:38,316 DEBUG [Thread-1512] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:54:38,316 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:54:38,316 DEBUG [Thread-1512] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:54:38,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:38,316 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:38,316 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:38,318 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:38,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:38,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(9158): Flush requested on 40cf532a35b33bde962f4f28e36e887b 2023-04-25 02:54:38,364 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 40cf532a35b33bde962f4f28e36e887b 3/3 column families, dataSize=143.13 KB heapSize=405.28 KB 2023-04-25 02:54:38,366 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1682391277747, max timestamp=1682391278363Num uniques 1018; 2023-04-25 02:54:38,366 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-25 02:54:38,366 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1682391277747, max timestamp=1682391278363Num uniques 1118; 2023-04-25 02:54:38,366 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1682391277747, max timestamp=1682391278363Num uniques 1118; 2023-04-25 02:54:38,366 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=51198, new segment data size=9978, suffix heap size=144240,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1118, new segment cells count=218 2023-04-25 02:54:38,366 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 40cf532a35b33bde962f4f28e36e887b, store=A 2023-04-25 02:54:38,366 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=51198, new segment data size=9978, suffix heap size=144240,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1118, new segment cells count=218 2023-04-25 02:54:38,366 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 02:54:38,367 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 40cf532a35b33bde962f4f28e36e887b, store=B 2023-04-25 02:54:38,367 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 02:54:38,367 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 40cf532a35b33bde962f4f28e36e887b, store=C 2023-04-25 02:54:38,367 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 02:54:38,383 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682391278380, max timestamp=1682391278381Num uniques 859; 2023-04-25 02:54:38,383 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-25 02:54:38,383 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682391278380, max timestamp=1682391278381Num uniques 859; 2023-04-25 02:54:38,383 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-25 02:54:38,383 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682391278380, max timestamp=1682391278381Num uniques 859; 2023-04-25 02:54:38,383 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-25 02:54:38,400 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1682391278388, max timestamp=1682391278398Num uniques 1018; 2023-04-25 02:54:38,400 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-25 02:54:38,400 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1682391278388, max timestamp=1682391278398Num uniques 1018; 2023-04-25 02:54:38,400 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-25 02:54:38,400 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1682391278388, max timestamp=1682391278398Num uniques 1018; 2023-04-25 02:54:38,400 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-25 02:54:38,416 DEBUG [Thread-1512] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:54:38,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:54:38,418 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:54:38,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:54:38,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:54:38,424 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:54:38,424 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:38,424 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:54:38,424 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:54:38,424 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:38,424 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:38,426 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:38,426 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:38,426 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:38,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:38,426 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:38,426 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:54:38,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:38,426 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:38,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:54:38,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:38,427 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:54:38,427 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:54:38,427 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:54:38,427 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:54:38,427 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:38,427 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:54:38,427 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:54:38,427 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:54:38,427 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. 2023-04-25 02:54:38,427 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. started... 2023-04-25 02:54:38,427 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. as already flushing 2023-04-25 02:54:38,427 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. 2023-04-25 02:54:38,427 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:54:38,427 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:54:38,428 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:38,428 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:54:38,428 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:54:38,430 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:38,430 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:38,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:38,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:38,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:38,430 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:38,430 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:54:38,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:38,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:38,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:38,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:38,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:38,431 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:54:38,431 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:54:38,431 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5f9f7576[Count = 0] remaining members to acquire global barrier 2023-04-25 02:54:38,431 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:38,433 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:38,433 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:38,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:38,433 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:54:38,433 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:54:38,433 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:54:38,433 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:38,433 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:54:38,434 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:54:38,434 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:38,434 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:38,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:38,434 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:54:38,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:38,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:38,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:38,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:38,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:38,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:38,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:38,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:38,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:38,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:54:38,436 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:54:38,436 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:54:38,436 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:54:38,436 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:54:38,436 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:54:38,436 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:54:38,438 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10382, totalHeapSize=23848, min timestamp=1682391278405, max timestamp=1682391278414Num uniques 1027; 2023-04-25 02:54:38,438 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=47022, new segment data size=10382, suffix heap size=134776,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=227 2023-04-25 02:54:38,439 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10382, totalHeapSize=23848, min timestamp=1682391278405, max timestamp=1682391278414Num uniques 1027; 2023-04-25 02:54:38,439 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=47022, new segment data size=10382, suffix heap size=134776,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=227 2023-04-25 02:54:38,439 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10382, totalHeapSize=23848, min timestamp=1682391278405, max timestamp=1682391278414Num uniques 1027; 2023-04-25 02:54:38,439 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=47022, new segment data size=10382, suffix heap size=134776,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=227 2023-04-25 02:54:38,440 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:38,440 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:38,440 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:38,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:38,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:38,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:38,440 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:38,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:38,441 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:38,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:38,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:38,441 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:38,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:38,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:38,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:38,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:38,442 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:38,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:38,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:38,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:38,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:38,445 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:38,445 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:38,446 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:38,446 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:38,446 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:38,446 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:38,446 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:38,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:38,446 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:38,446 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:38,446 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:38,446 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:38,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:38,446 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:54:38,446 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:54:38,446 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:54:38,446 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:38,447 DEBUG [Thread-1512] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:54:38,447 DEBUG [Thread-1512] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:54:38,455 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=286, cellSize=13096, totalHeapSize=29984, min timestamp=1682391278448, max timestamp=1682391278453Num uniques 1086; 2023-04-25 02:54:38,455 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=13096, suffix heap size=140912,new segment heap size=29984 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=286 2023-04-25 02:54:38,456 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=286, cellSize=13096, totalHeapSize=29984, min timestamp=1682391278448, max timestamp=1682391278453Num uniques 1086; 2023-04-25 02:54:38,456 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=13096, suffix heap size=140912,new segment heap size=29984 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=286 2023-04-25 02:54:38,456 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=286, cellSize=13096, totalHeapSize=29984, min timestamp=1682391278448, max timestamp=1682391278453Num uniques 1086; 2023-04-25 02:54:38,456 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=13096, suffix heap size=140912,new segment heap size=29984 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=286 2023-04-25 02:54:38,470 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:38,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59158 deadline: 1682391338467, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:38,471 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:38,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59214 deadline: 1682391338468, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:38,472 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:38,472 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59216 deadline: 1682391338469, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:38,472 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8920, totalHeapSize=20520, min timestamp=1682391278463, max timestamp=1682391278469Num uniques 1145; 2023-04-25 02:54:38,472 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=52430, new segment data size=8920, suffix heap size=147048,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1145, new segment cells count=195 2023-04-25 02:54:38,472 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8920, totalHeapSize=20520, min timestamp=1682391278463, max timestamp=1682391278469Num uniques 1145; 2023-04-25 02:54:38,472 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8920, totalHeapSize=20520, min timestamp=1682391278463, max timestamp=1682391278469Num uniques 1145; 2023-04-25 02:54:38,472 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=52430, new segment data size=8920, suffix heap size=147048,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1145, new segment cells count=195 2023-04-25 02:54:38,472 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=52430, new segment data size=8920, suffix heap size=147048,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1145, new segment cells count=195 2023-04-25 02:54:38,501 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9340, totalHeapSize=21456, min timestamp=1682391278488, max timestamp=1682391278498Num uniques 1054; 2023-04-25 02:54:38,501 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9340, totalHeapSize=21456, min timestamp=1682391278488, max timestamp=1682391278498Num uniques 1054; 2023-04-25 02:54:38,501 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9340, totalHeapSize=21456, min timestamp=1682391278488, max timestamp=1682391278498Num uniques 1054; 2023-04-25 02:54:38,501 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=9340, suffix heap size=137584,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=204 2023-04-25 02:54:38,501 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=9340, suffix heap size=137584,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=204 2023-04-25 02:54:38,501 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=9340, suffix heap size=137584,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=204 2023-04-25 02:54:38,530 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1682391278522, max timestamp=1682391278528Num uniques 1063; 2023-04-25 02:54:38,530 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=48678, new segment data size=7458, suffix heap size=138520,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1063, new segment cells count=163 2023-04-25 02:54:38,530 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1682391278522, max timestamp=1682391278528Num uniques 1063; 2023-04-25 02:54:38,530 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=48678, new segment data size=7458, suffix heap size=138520,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1063, new segment cells count=163 2023-04-25 02:54:38,531 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1682391278522, max timestamp=1682391278528Num uniques 1063; 2023-04-25 02:54:38,531 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=48678, new segment data size=7458, suffix heap size=138520,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1063, new segment cells count=163 2023-04-25 02:54:38,547 DEBUG [Thread-1512] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:54:38,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:54:38,549 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:54:38,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:54:38,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:54:38,550 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:54:38,550 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:38,550 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:54:38,550 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:54:38,550 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:38,550 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:38,551 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:38,551 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:38,551 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:38,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:38,551 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:38,551 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:54:38,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:38,552 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:38,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:54:38,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:38,552 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:54:38,552 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:54:38,553 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:54:38,553 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:54:38,553 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:38,553 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:54:38,553 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:54:38,553 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:54:38,553 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. 2023-04-25 02:54:38,553 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. started... 2023-04-25 02:54:38,553 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. as already flushing 2023-04-25 02:54:38,553 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. 2023-04-25 02:54:38,553 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:54:38,553 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:54:38,553 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:38,553 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:54:38,553 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:54:38,555 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:38,555 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:38,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:38,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:38,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:38,555 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:38,555 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:54:38,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:38,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:38,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:38,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:38,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:38,556 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:54:38,556 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@111bd54f[Count = 0] remaining members to acquire global barrier 2023-04-25 02:54:38,556 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:54:38,557 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:38,558 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:38,558 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:38,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:38,558 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:38,558 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:54:38,559 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:54:38,558 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:54:38,559 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:54:38,561 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:54:38,561 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:38,561 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:38,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:38,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:38,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:38,561 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:54:38,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:38,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:38,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:38,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:38,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:38,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:38,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:38,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:54:38,564 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:54:38,564 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:54:38,564 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:54:38,564 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:54:38,564 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:54:38,564 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:54:38,565 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:38,565 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:38,565 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:38,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:38,565 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:38,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:38,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:38,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:38,566 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:38,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:38,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:38,566 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:38,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:38,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:38,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:38,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:38,567 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:38,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:38,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:38,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:38,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:38,569 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:38,569 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:38,569 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:38,569 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:38,569 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:38,569 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:54:38,569 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:54:38,569 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:38,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:38,569 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:38,570 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:54:38,570 DEBUG [Thread-1512] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:54:38,570 DEBUG [Thread-1512] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:54:38,570 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:38,570 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:38,570 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:38,570 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:38,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:38,570 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:38,575 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7872, totalHeapSize=18128, min timestamp=1682391278541, max timestamp=1682391278573Num uniques 1022; 2023-04-25 02:54:38,575 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=7872, suffix heap size=134256,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=172 2023-04-25 02:54:38,576 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7872, totalHeapSize=18128, min timestamp=1682391278541, max timestamp=1682391278573Num uniques 1022; 2023-04-25 02:54:38,576 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=7872, suffix heap size=134256,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=172 2023-04-25 02:54:38,576 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7872, totalHeapSize=18128, min timestamp=1682391278541, max timestamp=1682391278573Num uniques 1022; 2023-04-25 02:54:38,576 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=7872, suffix heap size=134256,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=172 2023-04-25 02:54:38,588 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=281, cellSize=12866, totalHeapSize=29464, min timestamp=1682391278581, max timestamp=1682391278586Num uniques 1031; 2023-04-25 02:54:38,588 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=47216, new segment data size=12866, suffix heap size=135192,new segment heap size=29464 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1031, new segment cells count=281 2023-04-25 02:54:38,588 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=281, cellSize=12866, totalHeapSize=29464, min timestamp=1682391278581, max timestamp=1682391278586Num uniques 1031; 2023-04-25 02:54:38,588 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=281, cellSize=12866, totalHeapSize=29464, min timestamp=1682391278581, max timestamp=1682391278586Num uniques 1031; 2023-04-25 02:54:38,589 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=47216, new segment data size=12866, suffix heap size=135192,new segment heap size=29464 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1031, new segment cells count=281 2023-04-25 02:54:38,588 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=47216, new segment data size=12866, suffix heap size=135192,new segment heap size=29464 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1031, new segment cells count=281 2023-04-25 02:54:38,604 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=190, cellSize=8700, totalHeapSize=20000, min timestamp=1682391278599, max timestamp=1682391278602Num uniques 1140; 2023-04-25 02:54:38,604 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=52210, new segment data size=8700, suffix heap size=146528,new segment heap size=20000 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1140, new segment cells count=190 2023-04-25 02:54:38,605 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=190, cellSize=8700, totalHeapSize=20000, min timestamp=1682391278599, max timestamp=1682391278602Num uniques 1140; 2023-04-25 02:54:38,605 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=190, cellSize=8690, totalHeapSize=20000, min timestamp=1682391278599, max timestamp=1682391278602Num uniques 1140; 2023-04-25 02:54:38,605 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=52210, new segment data size=8700, suffix heap size=146528,new segment heap size=20000 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1140, new segment cells count=190 2023-04-25 02:54:38,605 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=52200, new segment data size=8690, suffix heap size=146528,new segment heap size=20000 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1140, new segment cells count=190 2023-04-25 02:54:38,618 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=299, cellSize=13694, totalHeapSize=31336, min timestamp=1682391278610, max timestamp=1682391278615Num uniques 1049; 2023-04-25 02:54:38,618 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=48044, new segment data size=13694, suffix heap size=137064,new segment heap size=31336 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1049, new segment cells count=299 2023-04-25 02:54:38,618 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=299, cellSize=13694, totalHeapSize=31336, min timestamp=1682391278610, max timestamp=1682391278615Num uniques 1049; 2023-04-25 02:54:38,618 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=299, cellSize=13694, totalHeapSize=31336, min timestamp=1682391278610, max timestamp=1682391278615Num uniques 1049; 2023-04-25 02:54:38,618 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=48044, new segment data size=13694, suffix heap size=137064,new segment heap size=31336 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1049, new segment cells count=299 2023-04-25 02:54:38,618 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=48044, new segment data size=13694, suffix heap size=137064,new segment heap size=31336 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1049, new segment cells count=299 2023-04-25 02:54:38,631 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:38,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59214 deadline: 1682391338628, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:38,632 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:38,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59160 deadline: 1682391338629, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:38,633 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:38,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59186 deadline: 1682391338630, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:38,633 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:38,634 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59216 deadline: 1682391338630, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:38,634 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:38,634 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59158 deadline: 1682391338631, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:38,670 DEBUG [Thread-1512] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:54:38,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:54:38,672 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:54:38,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:54:38,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:54:38,673 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:54:38,673 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:38,673 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:54:38,673 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:54:38,674 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:38,674 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:38,675 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:38,675 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:38,675 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:38,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:38,676 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:38,676 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:54:38,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:38,676 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:38,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:54:38,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:38,676 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:54:38,677 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:54:38,677 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:54:38,677 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:54:38,677 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:38,677 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:54:38,677 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:54:38,677 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:54:38,677 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. 2023-04-25 02:54:38,677 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. started... 2023-04-25 02:54:38,677 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. as already flushing 2023-04-25 02:54:38,677 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. 2023-04-25 02:54:38,677 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:54:38,677 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:54:38,677 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:38,677 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:54:38,678 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:54:38,680 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:38,680 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:38,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:38,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:38,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:38,681 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:38,681 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:54:38,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:38,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:38,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:38,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:38,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:38,682 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:54:38,682 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:54:38,682 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4fcef871[Count = 0] remaining members to acquire global barrier 2023-04-25 02:54:38,682 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:38,683 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:38,683 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:38,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:38,683 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:54:38,683 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:54:38,683 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:54:38,683 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:38,683 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:54:38,685 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:38,685 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:54:38,685 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:38,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:38,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:38,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:38,685 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:54:38,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:38,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:38,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:38,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:38,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:38,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:38,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:38,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:54:38,687 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:54:38,687 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:54:38,687 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:54:38,687 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:54:38,687 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:54:38,687 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:54:38,689 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:38,689 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:38,689 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:38,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:38,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:38,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:38,689 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:38,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:38,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:38,690 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:38,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:38,690 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:38,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:38,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:38,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:38,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:38,691 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:38,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:38,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:38,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:38,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:38,698 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:38,698 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:38,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:38,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:38,698 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:38,698 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:38,698 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:38,698 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:38,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:38,698 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:54:38,698 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:38,698 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:38,699 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:38,699 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:38,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:38,699 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:54:38,699 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:54:38,699 DEBUG [Thread-1512] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:54:38,699 DEBUG [Thread-1512] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:54:38,733 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:38,733 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59214 deadline: 1682391338732, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:38,733 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:38,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59160 deadline: 1682391338733, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:38,736 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:38,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59186 deadline: 1682391338734, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:38,736 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:38,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59216 deadline: 1682391338734, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:38,736 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:38,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59158 deadline: 1682391338734, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:38,777 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=50 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp/A/eeace9228e5749bca7c0ad6499f5263e 2023-04-25 02:54:38,799 DEBUG [Thread-1512] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:54:38,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:54:38,801 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:54:38,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:54:38,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:54:38,803 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:54:38,803 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:38,803 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:54:38,803 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:54:38,803 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:38,803 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:38,804 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:38,804 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:38,804 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:38,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:38,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:38,805 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:38,805 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:54:38,805 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:38,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:54:38,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:38,805 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:54:38,806 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:54:38,806 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:54:38,806 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:54:38,806 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:38,806 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:54:38,806 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:54:38,806 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:54:38,806 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. 2023-04-25 02:54:38,806 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. started... 2023-04-25 02:54:38,806 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. as already flushing 2023-04-25 02:54:38,806 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. 2023-04-25 02:54:38,806 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:54:38,807 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:54:38,807 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:38,807 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:54:38,807 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:54:38,810 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:38,810 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:38,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:38,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:38,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:38,811 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:38,811 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:54:38,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:38,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:38,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:38,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:38,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:38,812 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:54:38,812 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:54:38,812 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@25a4d205[Count = 0] remaining members to acquire global barrier 2023-04-25 02:54:38,812 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:38,814 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:38,814 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:38,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:38,814 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:38,815 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:54:38,814 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:54:38,815 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:54:38,815 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:54:38,816 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:54:38,816 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:38,816 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:38,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:38,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:38,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:38,816 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:54:38,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:38,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:38,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:38,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:38,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:38,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:38,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:38,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:54:38,818 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:54:38,818 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:54:38,819 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:54:38,819 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:54:38,819 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:54:38,819 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:54:38,820 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:38,820 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:38,820 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:38,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:38,820 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:38,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:38,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:38,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:38,821 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:38,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:38,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:38,821 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:38,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:38,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:38,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:38,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:38,822 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:38,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:38,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:38,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:38,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:38,824 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:38,824 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:38,824 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:38,824 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:38,825 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:38,824 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:38,825 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:54:38,825 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:54:38,825 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:38,825 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:38,825 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:38,825 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:38,825 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:38,825 DEBUG [Thread-1512] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:54:38,825 DEBUG [Thread-1512] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:54:38,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:38,826 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:38,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:38,925 DEBUG [Thread-1512] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:54:38,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:54:38,928 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:54:38,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:54:38,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:54:38,929 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:54:38,929 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:38,929 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:54:38,929 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:54:38,929 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:38,929 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:38,931 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:38,931 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:38,932 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:38,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:38,932 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:38,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:38,932 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:54:38,932 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:38,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:54:38,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:38,933 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:54:38,933 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:54:38,933 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:54:38,933 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:54:38,933 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:38,933 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:54:38,933 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:54:38,933 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:54:38,933 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. 2023-04-25 02:54:38,933 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. started... 2023-04-25 02:54:38,934 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. as already flushing 2023-04-25 02:54:38,934 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. 2023-04-25 02:54:38,934 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:54:38,934 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:54:38,934 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:38,934 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:54:38,934 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:54:38,935 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:38,935 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:38,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:38,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:38,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:38,935 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:38,936 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:54:38,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:38,936 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:38,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:38,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59160 deadline: 1682391338934, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:38,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:38,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:38,937 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:38,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:38,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59214 deadline: 1682391338935, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:38,937 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:54:38,937 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1b43702[Count = 0] remaining members to acquire global barrier 2023-04-25 02:54:38,937 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:54:38,937 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:38,938 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:38,938 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:38,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59186 deadline: 1682391338937, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:38,939 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:38,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:38,939 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:54:38,939 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:54:38,939 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:54:38,939 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:38,939 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:54:38,939 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:38,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59216 deadline: 1682391338938, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:38,940 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:38,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59158 deadline: 1682391338938, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:38,940 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:38,940 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:54:38,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:38,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:38,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:38,940 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:38,940 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:54:38,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:38,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:38,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:38,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:38,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:38,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:38,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:38,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:54:38,942 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:54:38,942 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:54:38,942 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:54:38,943 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:54:38,943 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:54:38,943 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:54:38,945 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:38,945 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:38,945 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:38,945 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:38,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:38,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:38,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:38,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:38,945 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:38,945 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:38,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:38,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:38,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:38,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:38,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:38,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:38,947 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:38,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:38,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:38,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:38,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:38,949 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:38,949 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:38,949 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:54:38,949 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:54:38,949 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:38,949 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:38,949 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:38,949 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:38,949 DEBUG [Thread-1512] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:54:38,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:38,949 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:38,949 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:38,949 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:38,949 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:38,949 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:38,949 DEBUG [Thread-1512] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:54:38,950 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:54:38,950 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:38,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:39,050 DEBUG [Thread-1512] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:54:39,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:54:39,052 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:54:39,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:54:39,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:54:39,053 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:54:39,053 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:39,053 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:54:39,053 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:54:39,053 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:39,053 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:39,054 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:39,054 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:39,054 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:39,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:39,055 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:39,055 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:54:39,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:39,055 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:39,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:54:39,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:39,056 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:54:39,056 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:54:39,056 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:54:39,056 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:54:39,056 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:39,056 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:54:39,056 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:54:39,056 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:54:39,056 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. 2023-04-25 02:54:39,056 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. started... 2023-04-25 02:54:39,056 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. as already flushing 2023-04-25 02:54:39,057 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. 2023-04-25 02:54:39,057 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:54:39,057 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:54:39,057 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:39,057 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:54:39,057 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:54:39,058 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:39,058 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:39,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:39,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:39,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:39,058 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:39,058 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:54:39,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:39,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:39,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:39,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:39,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:39,060 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:54:39,060 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4eca3bd6[Count = 0] remaining members to acquire global barrier 2023-04-25 02:54:39,060 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:54:39,060 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:39,062 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:39,062 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:39,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:39,062 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:54:39,062 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:39,062 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:54:39,062 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:54:39,062 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:54:39,063 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:39,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:39,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:39,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:39,063 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:54:39,063 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:39,063 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:54:39,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:39,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:39,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:39,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:39,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:39,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:39,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:39,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:54:39,066 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:54:39,066 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:54:39,066 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:54:39,066 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:54:39,066 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:54:39,066 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:54:39,067 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:39,067 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:39,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:39,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:39,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:39,067 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:39,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:39,067 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:39,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:39,068 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:39,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:39,067 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:39,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:39,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:39,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:39,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:39,070 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:39,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:39,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:39,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:39,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:39,071 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:39,071 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:39,071 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:39,072 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:39,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:39,072 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:39,072 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:39,072 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:39,072 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:39,072 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:39,072 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:39,072 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:54:39,072 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:54:39,072 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:54:39,072 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:39,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:39,072 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:39,072 DEBUG [Thread-1512] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:54:39,072 DEBUG [Thread-1512] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:54:39,173 DEBUG [Thread-1512] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:54:39,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:54:39,175 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:54:39,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:54:39,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:54:39,176 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:54:39,176 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:39,176 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:54:39,176 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:54:39,176 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:39,176 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:39,181 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:39,181 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:39,181 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:39,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:39,182 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:39,182 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:54:39,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:39,182 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:39,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:54:39,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:39,182 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:54:39,183 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:54:39,183 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:54:39,183 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:54:39,183 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:39,183 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:54:39,183 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:54:39,183 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:54:39,183 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. 2023-04-25 02:54:39,183 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. started... 2023-04-25 02:54:39,183 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. as already flushing 2023-04-25 02:54:39,183 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. 2023-04-25 02:54:39,183 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:54:39,183 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:54:39,183 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:39,183 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:54:39,183 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:54:39,185 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:39,185 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:39,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:39,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:39,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:39,185 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:39,185 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:54:39,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:39,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:39,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:39,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:39,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:39,186 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:54:39,186 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@76ff4f0e[Count = 0] remaining members to acquire global barrier 2023-04-25 02:54:39,186 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:54:39,186 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:39,187 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:39,187 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:39,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:39,188 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:54:39,188 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:54:39,188 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:54:39,188 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:39,188 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:54:39,189 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=50 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp/B/f11aeab20c8b4454bc900c2fa1d8ad0c 2023-04-25 02:54:39,190 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:39,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:39,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:39,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:39,190 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:54:39,190 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:39,190 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:54:39,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:39,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:39,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:39,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:39,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:39,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:39,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:39,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:54:39,193 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:54:39,193 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:54:39,193 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:54:39,193 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:54:39,193 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:54:39,193 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:54:39,194 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:39,194 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:39,194 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:39,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:39,195 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:39,195 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:39,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:39,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:39,196 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:39,198 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:39,198 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:39,198 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:39,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:39,198 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:39,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:39,199 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:39,199 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:54:39,199 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:54:39,199 DEBUG [Thread-1512] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:54:39,199 DEBUG [Thread-1512] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:54:39,200 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:39,200 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:39,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:39,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:39,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:39,200 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:39,200 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:39,200 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:39,200 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:39,200 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:39,200 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:39,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:39,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:39,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:39,201 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:54:39,238 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:39,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59160 deadline: 1682391339237, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:39,239 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:39,239 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59214 deadline: 1682391339238, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:39,242 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:39,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59186 deadline: 1682391339240, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:39,242 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:39,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59216 deadline: 1682391339240, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:39,244 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:39,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59158 deadline: 1682391339243, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:39,299 DEBUG [Thread-1512] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:54:39,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:54:39,301 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:54:39,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:54:39,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:54:39,302 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:54:39,303 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:39,303 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:54:39,303 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:54:39,303 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:39,303 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:39,304 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:39,304 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:39,304 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:39,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:39,304 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:39,305 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:54:39,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:39,305 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:39,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:54:39,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:39,305 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:54:39,305 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:54:39,306 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:54:39,306 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:54:39,306 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:39,306 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:54:39,306 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:54:39,307 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:54:39,307 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. 2023-04-25 02:54:39,307 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. started... 2023-04-25 02:54:39,307 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. as already flushing 2023-04-25 02:54:39,307 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. 2023-04-25 02:54:39,307 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:54:39,307 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:54:39,307 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:39,307 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:54:39,307 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:54:39,309 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:39,309 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:39,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:39,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:39,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:39,310 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:39,310 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:54:39,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:39,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:39,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:39,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:39,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:39,311 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:54:39,311 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:54:39,311 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@75488f2e[Count = 0] remaining members to acquire global barrier 2023-04-25 02:54:39,311 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:39,312 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:39,312 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:39,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:39,312 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:54:39,312 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:54:39,312 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:54:39,312 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:39,312 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:54:39,313 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:54:39,314 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:39,314 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:39,314 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:54:39,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:39,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:39,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:39,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:39,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:39,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:39,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:39,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:39,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:39,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:39,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:54:39,316 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:54:39,316 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:54:39,316 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:54:39,316 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:54:39,316 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:54:39,316 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:54:39,318 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:39,318 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:39,318 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:39,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:39,318 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:39,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:39,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:39,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:39,318 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:39,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:39,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:39,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:39,318 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:39,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:39,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:39,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:39,320 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:39,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:39,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:39,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:39,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:39,323 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:39,323 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:39,323 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:39,323 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:39,323 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:54:39,323 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:39,323 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:54:39,323 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:39,323 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:39,323 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:39,323 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:39,323 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:39,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:39,323 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:54:39,323 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:39,324 DEBUG [Thread-1512] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:54:39,324 DEBUG [Thread-1512] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:54:39,324 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:39,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:39,424 DEBUG [Thread-1512] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:54:39,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:54:39,426 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:54:39,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:54:39,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:54:39,428 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:54:39,428 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:39,428 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:54:39,428 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:54:39,428 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:39,428 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:39,429 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:39,429 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:39,429 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:39,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:39,430 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:39,430 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:54:39,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:39,430 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:39,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:54:39,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:39,430 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:54:39,431 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:54:39,431 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:54:39,431 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:54:39,431 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:39,431 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:54:39,431 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:54:39,431 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:54:39,431 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. 2023-04-25 02:54:39,431 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. started... 2023-04-25 02:54:39,431 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. as already flushing 2023-04-25 02:54:39,431 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. 2023-04-25 02:54:39,431 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:54:39,431 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:54:39,431 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:39,432 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:54:39,432 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:54:39,433 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:39,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:39,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:39,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:39,433 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:39,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:39,433 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:39,433 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:54:39,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:39,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:39,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:39,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:39,435 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:54:39,435 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5564baa2[Count = 0] remaining members to acquire global barrier 2023-04-25 02:54:39,435 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:54:39,435 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:39,436 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:39,436 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:39,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:39,436 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:54:39,436 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:54:39,436 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:54:39,436 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:39,436 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:54:39,438 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:39,438 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:54:39,438 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:39,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:39,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:39,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:39,438 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:54:39,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:39,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:39,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:39,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:39,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:39,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:39,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:39,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:54:39,441 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:54:39,441 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:54:39,441 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:54:39,441 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:54:39,441 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:54:39,441 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:54:39,442 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:39,442 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:39,442 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:39,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:39,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:39,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:39,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:39,442 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:39,443 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:39,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:39,443 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:39,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:39,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:39,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:39,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:39,444 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:39,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:39,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:39,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:39,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:39,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:39,446 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:39,446 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:39,446 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:39,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:39,446 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:39,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:39,447 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:39,447 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:39,447 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:39,447 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:39,447 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:39,447 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:54:39,447 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:54:39,447 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:39,447 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:39,447 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:39,447 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:54:39,447 DEBUG [Thread-1512] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:54:39,447 DEBUG [Thread-1512] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:54:39,547 DEBUG [Thread-1512] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:54:39,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:54:39,549 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:54:39,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:54:39,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:54:39,551 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:54:39,551 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:39,551 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:54:39,551 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:54:39,551 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:39,551 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:39,552 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:39,552 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:39,552 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:39,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:39,553 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:39,553 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:54:39,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:39,553 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:39,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:54:39,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:39,553 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:54:39,554 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:54:39,554 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:54:39,554 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:54:39,554 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:39,554 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:54:39,554 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:54:39,554 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:54:39,554 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. 2023-04-25 02:54:39,554 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. started... 2023-04-25 02:54:39,554 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. as already flushing 2023-04-25 02:54:39,554 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. 2023-04-25 02:54:39,554 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:54:39,555 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:54:39,555 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:39,555 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:54:39,555 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:54:39,559 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:39,559 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:39,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:39,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:39,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:39,559 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:39,559 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:54:39,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:39,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:39,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:39,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:39,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:39,561 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:54:39,561 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4a6f3f53[Count = 0] remaining members to acquire global barrier 2023-04-25 02:54:39,561 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:54:39,561 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:39,562 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:39,562 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:39,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:39,562 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:39,562 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:54:39,562 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:54:39,562 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:54:39,562 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:54:39,564 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:54:39,564 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:39,564 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:39,564 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:54:39,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:39,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:39,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:39,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:39,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:39,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:39,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:39,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:39,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:39,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:39,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:54:39,568 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:54:39,568 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:54:39,568 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:54:39,568 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:54:39,568 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:54:39,568 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:54:39,571 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:39,571 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:39,571 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:39,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:39,571 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:39,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:39,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:39,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:39,571 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:39,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:39,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:39,572 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:39,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:39,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:39,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:39,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:39,573 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:39,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:39,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:39,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:39,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:39,575 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:39,575 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:39,575 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:39,575 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:39,575 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:39,575 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:54:39,575 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:39,575 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:54:39,575 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:54:39,575 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:39,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:39,575 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:39,576 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:39,576 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:39,576 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:39,576 DEBUG [Thread-1512] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:54:39,576 DEBUG [Thread-1512] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:54:39,576 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:39,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:39,610 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=50 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp/C/621545d991884cd6bda0d135d0fbfff9 2023-04-25 02:54:39,617 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp/A/eeace9228e5749bca7c0ad6499f5263e as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A/eeace9228e5749bca7c0ad6499f5263e 2023-04-25 02:54:39,621 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A/eeace9228e5749bca7c0ad6499f5263e, entries=150, sequenceid=50, filesize=11.6 K 2023-04-25 02:54:39,622 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp/B/f11aeab20c8b4454bc900c2fa1d8ad0c as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B/f11aeab20c8b4454bc900c2fa1d8ad0c 2023-04-25 02:54:39,627 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B/f11aeab20c8b4454bc900c2fa1d8ad0c, entries=150, sequenceid=50, filesize=11.6 K 2023-04-25 02:54:39,628 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp/C/621545d991884cd6bda0d135d0fbfff9 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C/621545d991884cd6bda0d135d0fbfff9 2023-04-25 02:54:39,637 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C/621545d991884cd6bda0d135d0fbfff9, entries=150, sequenceid=50, filesize=11.6 K 2023-04-25 02:54:39,638 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~80.58 KB/82512, currentSize=154.31 KB/158010 for 40cf532a35b33bde962f4f28e36e887b in 1274ms, sequenceid=50, compaction requested=false 2023-04-25 02:54:39,638 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 40cf532a35b33bde962f4f28e36e887b: 2023-04-25 02:54:39,676 DEBUG [Thread-1512] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:54:39,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:54:39,678 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:54:39,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:54:39,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:54:39,679 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:54:39,679 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:39,679 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:54:39,679 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:54:39,680 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:39,680 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:39,681 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:39,681 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:39,681 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:39,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:39,681 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:39,681 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:54:39,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:39,682 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:39,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:54:39,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:39,682 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:54:39,682 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:54:39,682 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:54:39,682 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:54:39,682 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:39,682 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:54:39,683 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:54:39,683 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:54:39,683 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. 2023-04-25 02:54:39,683 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. started... 2023-04-25 02:54:39,683 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 40cf532a35b33bde962f4f28e36e887b 3/3 column families, dataSize=154.31 KB heapSize=431.63 KB 2023-04-25 02:54:39,683 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 40cf532a35b33bde962f4f28e36e887b, store=A 2023-04-25 02:54:39,683 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 02:54:39,683 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 40cf532a35b33bde962f4f28e36e887b, store=B 2023-04-25 02:54:39,683 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 02:54:39,683 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 40cf532a35b33bde962f4f28e36e887b, store=C 2023-04-25 02:54:39,683 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 02:54:39,691 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=259 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp/A/56fdb1660bae4bcd935540ba6f54b811 2023-04-25 02:54:39,701 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=259 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp/B/b373fbd5f50c43c1834fd6842d856498 2023-04-25 02:54:39,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(9158): Flush requested on 40cf532a35b33bde962f4f28e36e887b 2023-04-25 02:54:39,743 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. as already flushing 2023-04-25 02:54:39,746 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:39,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59214 deadline: 1682391339744, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:39,747 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:39,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59158 deadline: 1682391339745, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:39,748 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:39,748 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59186 deadline: 1682391339746, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:39,749 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:39,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59160 deadline: 1682391339747, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:39,749 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:39,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59216 deadline: 1682391339747, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:39,850 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:39,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59214 deadline: 1682391339848, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:39,851 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:39,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59186 deadline: 1682391339849, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:39,851 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:39,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59160 deadline: 1682391339850, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:39,851 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:39,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59216 deadline: 1682391339850, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:40,052 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:40,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59214 deadline: 1682391340052, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:40,054 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:40,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59216 deadline: 1682391340052, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:40,055 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:40,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59186 deadline: 1682391340053, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:40,055 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:40,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59160 deadline: 1682391340053, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:40,114 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=259 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp/C/47f6457404874cc5943e7bcb2e8c22c1 2023-04-25 02:54:40,119 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp/A/56fdb1660bae4bcd935540ba6f54b811 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A/56fdb1660bae4bcd935540ba6f54b811 2023-04-25 02:54:40,123 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A/56fdb1660bae4bcd935540ba6f54b811, entries=150, sequenceid=259, filesize=11.8 K 2023-04-25 02:54:40,124 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp/B/b373fbd5f50c43c1834fd6842d856498 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B/b373fbd5f50c43c1834fd6842d856498 2023-04-25 02:54:40,128 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B/b373fbd5f50c43c1834fd6842d856498, entries=150, sequenceid=259, filesize=11.8 K 2023-04-25 02:54:40,129 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp/C/47f6457404874cc5943e7bcb2e8c22c1 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C/47f6457404874cc5943e7bcb2e8c22c1 2023-04-25 02:54:40,133 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C/47f6457404874cc5943e7bcb2e8c22c1, entries=150, sequenceid=259, filesize=11.8 K 2023-04-25 02:54:40,134 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~154.31 KB/158010, heapSize ~431.58 KB/441936, currentSize=33.54 KB/34350 for 40cf532a35b33bde962f4f28e36e887b in 451ms, sequenceid=259, compaction requested=true 2023-04-25 02:54:40,135 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 40cf532a35b33bde962f4f28e36e887b: 2023-04-25 02:54:40,135 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. 2023-04-25 02:54:40,135 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:54:40,135 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:54:40,135 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:40,135 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:54:40,135 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:54:40,136 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:40,136 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:40,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:40,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:40,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:40,137 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:40,137 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:54:40,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:40,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:40,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:40,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:40,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:40,139 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:54:40,139 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@46649e2f[Count = 0] remaining members to acquire global barrier 2023-04-25 02:54:40,139 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:54:40,139 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:40,140 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:40,140 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:40,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:40,140 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:54:40,140 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:54:40,140 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:54:40,140 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:40,141 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:54:40,141 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:54:40,141 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:40,142 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:54:40,141 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:40,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:40,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:40,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:40,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:40,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:40,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:40,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:40,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:40,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:40,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:40,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:54:40,145 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:54:40,145 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:54:40,145 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:54:40,145 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:54:40,145 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:54:40,145 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:54:40,147 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:40,147 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:40,147 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:40,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:40,147 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:40,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:40,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:40,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:40,148 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:40,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:40,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:40,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:40,148 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:40,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:40,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:40,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:40,150 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:40,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:40,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:40,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:40,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:40,152 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:40,152 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:40,152 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:40,152 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:40,152 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:54:40,152 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:54:40,152 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:40,152 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:40,152 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:40,152 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:54:40,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:40,152 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:40,152 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:40,152 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:40,153 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:40,153 DEBUG [Thread-1512] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:54:40,153 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:40,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:40,153 DEBUG [Thread-1512] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:54:40,253 DEBUG [Thread-1512] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:54:40,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:54:40,255 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:54:40,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:54:40,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:54:40,257 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:54:40,258 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:40,258 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:54:40,258 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:54:40,258 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:40,258 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:40,259 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:40,259 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:40,259 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:40,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:40,259 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:40,259 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:54:40,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:40,260 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:40,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:54:40,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:40,260 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:54:40,260 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:54:40,260 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:54:40,261 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:54:40,261 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:40,261 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:54:40,261 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:54:40,261 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:54:40,261 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. 2023-04-25 02:54:40,261 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. started... 2023-04-25 02:54:40,261 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 40cf532a35b33bde962f4f28e36e887b 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-04-25 02:54:40,261 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 40cf532a35b33bde962f4f28e36e887b, store=A 2023-04-25 02:54:40,261 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:54:40,262 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 40cf532a35b33bde962f4f28e36e887b, store=B 2023-04-25 02:54:40,262 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:54:40,262 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 40cf532a35b33bde962f4f28e36e887b, store=C 2023-04-25 02:54:40,262 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:54:40,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(9158): Flush requested on 40cf532a35b33bde962f4f28e36e887b 2023-04-25 02:54:40,357 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. as already flushing 2023-04-25 02:54:40,397 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682391280389, max timestamp=1682391280395Num uniques 859; 2023-04-25 02:54:40,397 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682391280389, max timestamp=1682391280395Num uniques 859; 2023-04-25 02:54:40,397 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-25 02:54:40,397 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-25 02:54:40,398 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682391280389, max timestamp=1682391280395Num uniques 859; 2023-04-25 02:54:40,398 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-25 02:54:40,419 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=318, cellSize=14558, totalHeapSize=33312, min timestamp=1682391280405, max timestamp=1682391280416Num uniques 1118; 2023-04-25 02:54:40,419 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=318, cellSize=14558, totalHeapSize=33312, min timestamp=1682391280405, max timestamp=1682391280416Num uniques 1118; 2023-04-25 02:54:40,419 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=318, cellSize=14558, totalHeapSize=33312, min timestamp=1682391280405, max timestamp=1682391280416Num uniques 1118; 2023-04-25 02:54:40,419 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=51198, new segment data size=14558, suffix heap size=144240,new segment heap size=33312 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1118, new segment cells count=318 2023-04-25 02:54:40,419 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=51198, new segment data size=14558, suffix heap size=144240,new segment heap size=33312 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1118, new segment cells count=318 2023-04-25 02:54:40,419 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=51198, new segment data size=14558, suffix heap size=144240,new segment heap size=33312 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1118, new segment cells count=318 2023-04-25 02:54:40,438 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:40,438 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:40,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59216 deadline: 1682391340434, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:40,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59160 deadline: 1682391340435, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:40,438 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:40,438 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:40,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59214 deadline: 1682391340435, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:40,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59186 deadline: 1682391340435, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:40,541 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:40,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59160 deadline: 1682391340539, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:40,541 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:40,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59216 deadline: 1682391340539, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:40,541 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:40,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59214 deadline: 1682391340539, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:40,541 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:40,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59186 deadline: 1682391340539, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:40,674 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=267 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp/A/0104577a6bdf452cae48db5ff23270f6 2023-04-25 02:54:40,743 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:40,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59160 deadline: 1682391340742, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:40,743 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:40,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59186 deadline: 1682391340742, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:40,744 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:40,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59214 deadline: 1682391340742, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:40,744 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:40,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59216 deadline: 1682391340743, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:40,746 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-25 02:54:40,751 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:40,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59158 deadline: 1682391340749, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:41,045 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:41,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59160 deadline: 1682391341044, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:41,047 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:41,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59216 deadline: 1682391341045, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:41,047 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:41,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59186 deadline: 1682391341045, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:41,047 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:41,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59214 deadline: 1682391341046, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:41,085 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=267 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp/B/f5926eca392e48c6956392422ed54275 2023-04-25 02:54:41,098 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=267 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp/C/3f6bb3b42dc8411a848d42d32daecea6 2023-04-25 02:54:41,103 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp/A/0104577a6bdf452cae48db5ff23270f6 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A/0104577a6bdf452cae48db5ff23270f6 2023-04-25 02:54:41,112 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A/0104577a6bdf452cae48db5ff23270f6, entries=100, sequenceid=267, filesize=9.5 K 2023-04-25 02:54:41,113 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp/B/f5926eca392e48c6956392422ed54275 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B/f5926eca392e48c6956392422ed54275 2023-04-25 02:54:41,118 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B/f5926eca392e48c6956392422ed54275, entries=100, sequenceid=267, filesize=9.5 K 2023-04-25 02:54:41,119 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp/C/3f6bb3b42dc8411a848d42d32daecea6 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C/3f6bb3b42dc8411a848d42d32daecea6 2023-04-25 02:54:41,122 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C/3f6bb3b42dc8411a848d42d32daecea6, entries=100, sequenceid=267, filesize=9.5 K 2023-04-25 02:54:41,123 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=154.31 KB/158010 for 40cf532a35b33bde962f4f28e36e887b in 862ms, sequenceid=267, compaction requested=true 2023-04-25 02:54:41,123 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 40cf532a35b33bde962f4f28e36e887b: 2023-04-25 02:54:41,123 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. 2023-04-25 02:54:41,124 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:54:41,124 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:54:41,124 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:41,124 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:54:41,124 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:54:41,125 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:41,125 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:41,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:41,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:41,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:41,126 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:41,126 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:54:41,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:41,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:41,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:41,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:41,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:41,127 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:54:41,128 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@77e45832[Count = 0] remaining members to acquire global barrier 2023-04-25 02:54:41,128 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:54:41,128 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:41,136 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:41,136 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:41,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:41,136 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:54:41,136 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:54:41,136 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:54:41,137 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:41,137 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:54:41,138 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:41,138 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:54:41,138 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:41,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:41,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:41,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:41,138 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:54:41,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:41,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:41,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:41,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:41,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:41,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:41,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:41,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:54:41,141 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:54:41,141 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:54:41,141 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:54:41,141 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:54:41,141 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:54:41,141 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:54:41,143 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:41,143 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:41,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:41,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:41,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:41,143 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:41,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:41,143 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:41,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:41,143 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:41,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:41,143 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:41,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:41,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:41,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:41,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:41,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:41,145 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:41,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:41,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:41,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:41,147 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:41,147 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:41,147 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:41,147 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:41,147 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:41,147 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:54:41,147 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:41,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:41,147 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:54:41,147 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:54:41,147 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:41,147 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:41,147 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:41,147 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:41,147 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:41,147 DEBUG [Thread-1512] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:54:41,147 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:41,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:41,147 DEBUG [Thread-1512] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:54:41,248 DEBUG [Thread-1512] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:54:41,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:54:41,250 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:54:41,251 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:54:41,251 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:54:41,251 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:54:41,251 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:41,251 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:54:41,251 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:54:41,251 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:41,252 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:41,254 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:41,254 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:41,254 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:41,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:41,254 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:41,254 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:54:41,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:41,255 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:41,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:54:41,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:41,255 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:54:41,256 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:54:41,256 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:54:41,256 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:54:41,256 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:41,256 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:54:41,256 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:54:41,256 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:54:41,256 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. 2023-04-25 02:54:41,256 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. started... 2023-04-25 02:54:41,256 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 40cf532a35b33bde962f4f28e36e887b 3/3 column families, dataSize=154.31 KB heapSize=429.84 KB 2023-04-25 02:54:41,257 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 40cf532a35b33bde962f4f28e36e887b, store=A 2023-04-25 02:54:41,257 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 02:54:41,257 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 40cf532a35b33bde962f4f28e36e887b, store=B 2023-04-25 02:54:41,257 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 02:54:41,257 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 40cf532a35b33bde962f4f28e36e887b, store=C 2023-04-25 02:54:41,257 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 02:54:41,269 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=321 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp/A/3bec4c32bf214871996921c22931d36b 2023-04-25 02:54:41,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(9158): Flush requested on 40cf532a35b33bde962f4f28e36e887b 2023-04-25 02:54:41,551 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. as already flushing 2023-04-25 02:54:41,558 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:41,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59214 deadline: 1682391341553, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:41,558 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:41,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59160 deadline: 1682391341554, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:41,559 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:41,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59186 deadline: 1682391341555, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:41,561 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:41,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59216 deadline: 1682391341558, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:41,660 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:41,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59214 deadline: 1682391341659, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:41,660 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:41,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59160 deadline: 1682391341659, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:41,662 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:41,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59186 deadline: 1682391341660, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:41,664 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:41,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59216 deadline: 1682391341661, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:41,683 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=321 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp/B/cd0f47406b3e4b259e501639ad4f11c9 2023-04-25 02:54:41,701 DEBUG [regionserver/jenkins-hbase4:0.Chore.1] throttle.PressureAwareCompactionThroughputController(103): CompactionPressure is 0.07692307692307693, tune throughput to 53.85 MB/second 2023-04-25 02:54:41,728 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=321 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp/C/ab039aa53efc4464a6044bb43d22e721 2023-04-25 02:54:41,733 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp/A/3bec4c32bf214871996921c22931d36b as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A/3bec4c32bf214871996921c22931d36b 2023-04-25 02:54:41,738 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A/3bec4c32bf214871996921c22931d36b, entries=150, sequenceid=321, filesize=11.9 K 2023-04-25 02:54:41,738 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp/B/cd0f47406b3e4b259e501639ad4f11c9 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B/cd0f47406b3e4b259e501639ad4f11c9 2023-04-25 02:54:41,743 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B/cd0f47406b3e4b259e501639ad4f11c9, entries=150, sequenceid=321, filesize=11.9 K 2023-04-25 02:54:41,744 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp/C/ab039aa53efc4464a6044bb43d22e721 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C/ab039aa53efc4464a6044bb43d22e721 2023-04-25 02:54:41,757 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C/ab039aa53efc4464a6044bb43d22e721, entries=150, sequenceid=321, filesize=11.9 K 2023-04-25 02:54:41,758 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~154.31 KB/158010, heapSize ~429.80 KB/440112, currentSize=33.54 KB/34350 for 40cf532a35b33bde962f4f28e36e887b in 502ms, sequenceid=321, compaction requested=true 2023-04-25 02:54:41,758 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 40cf532a35b33bde962f4f28e36e887b: 2023-04-25 02:54:41,758 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. 2023-04-25 02:54:41,758 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:54:41,759 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:54:41,759 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:41,759 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:54:41,759 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:54:41,760 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:41,760 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:41,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:41,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:41,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:41,761 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:41,761 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:54:41,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:41,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:41,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:41,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:41,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:41,762 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:54:41,762 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7a55dc6[Count = 0] remaining members to acquire global barrier 2023-04-25 02:54:41,762 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:54:41,763 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:41,768 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:41,768 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:41,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:41,768 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:54:41,768 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:54:41,768 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:54:41,769 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:54:41,769 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:41,769 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:41,769 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:54:41,770 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:54:41,770 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:54:41,770 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:54:41,770 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:54:41,770 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:54:41,770 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:54:41,770 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:54:41,772 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:41,772 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:41,772 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:41,772 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:41,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:41,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:41,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:41,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:41,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:41,773 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:41,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:41,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:41,774 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:41,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:41,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:41,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:41,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:41,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:41,776 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:41,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:41,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:41,779 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:41,779 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:41,780 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:41,780 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:41,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:41,780 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:41,780 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:54:41,780 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:54:41,780 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:41,780 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:41,780 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:41,780 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:41,780 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:41,780 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:41,780 DEBUG [Thread-1512] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:54:41,780 DEBUG [Thread-1512] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:54:41,780 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:41,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:41,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(9158): Flush requested on 40cf532a35b33bde962f4f28e36e887b 2023-04-25 02:54:41,865 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 40cf532a35b33bde962f4f28e36e887b 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-25 02:54:41,866 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 40cf532a35b33bde962f4f28e36e887b, store=A 2023-04-25 02:54:41,866 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:54:41,866 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 40cf532a35b33bde962f4f28e36e887b, store=B 2023-04-25 02:54:41,866 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:54:41,866 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 40cf532a35b33bde962f4f28e36e887b, store=C 2023-04-25 02:54:41,866 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:54:41,875 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=332 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp/A/a5d3cbd452244da5a1c55147119f7e0a 2023-04-25 02:54:41,880 DEBUG [Thread-1512] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:54:41,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:54:41,883 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:54:41,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:54:41,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:54:41,890 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:54:41,890 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:41,890 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:54:41,891 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:54:41,891 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:41,891 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:41,892 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:41,892 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:41,892 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:41,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:41,893 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:41,893 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:54:41,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:41,893 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:41,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:54:41,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:41,893 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:54:41,893 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:54:41,893 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:54:41,894 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:54:41,894 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:41,894 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:54:41,894 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:54:41,894 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:54:41,894 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. 2023-04-25 02:54:41,894 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. started... 2023-04-25 02:54:41,894 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. as already flushing 2023-04-25 02:54:41,894 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. 2023-04-25 02:54:41,894 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:54:41,894 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:54:41,894 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:41,894 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:54:41,894 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:54:41,895 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:41,895 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:41,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:41,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:41,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:41,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:41,896 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:41,896 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:54:41,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:41,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:41,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:41,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:41,897 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:54:41,897 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@34836fc6[Count = 0] remaining members to acquire global barrier 2023-04-25 02:54:41,897 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:54:41,897 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:41,898 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:41,898 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:41,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:41,898 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:54:41,898 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:54:41,898 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:54:41,898 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:41,898 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:54:41,900 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:41,900 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:54:41,900 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:41,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:41,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:41,900 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:54:41,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:41,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:41,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:41,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:41,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:41,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:41,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:41,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:41,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:54:41,902 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:54:41,902 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:54:41,902 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:54:41,902 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:54:41,902 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:54:41,902 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:54:41,904 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:41,904 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:41,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:41,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:41,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:41,904 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:41,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:41,904 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:41,904 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:41,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:41,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:41,904 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:41,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:41,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:41,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:41,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:41,906 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:41,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:41,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:41,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:41,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:41,908 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:41,908 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:41,908 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:41,908 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:41,908 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:41,908 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:54:41,908 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:41,908 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:54:41,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:41,908 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:41,908 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:41,908 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:41,908 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:41,908 DEBUG [Thread-1512] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:54:41,908 DEBUG [Thread-1512] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:54:41,908 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:41,908 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:41,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:41,918 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:41,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59214 deadline: 1682391341916, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:41,919 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682391281911, max timestamp=1682391281917Num uniques 859; 2023-04-25 02:54:41,919 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682391281911, max timestamp=1682391281917Num uniques 859; 2023-04-25 02:54:41,919 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-25 02:54:41,920 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-25 02:54:41,920 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682391281911, max timestamp=1682391281917Num uniques 859; 2023-04-25 02:54:41,920 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-25 02:54:41,942 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:41,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59160 deadline: 1682391341937, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:41,942 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:41,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59186 deadline: 1682391341940, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:41,946 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:41,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59216 deadline: 1682391341942, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:42,008 DEBUG [Thread-1512] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:54:42,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:54:42,010 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:54:42,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:54:42,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:54:42,012 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:54:42,012 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:42,012 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:54:42,012 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:54:42,012 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:42,012 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:42,013 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:42,013 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:42,013 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:42,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:42,014 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:42,014 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:54:42,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:42,014 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:42,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:54:42,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:42,014 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:54:42,015 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:54:42,015 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:54:42,015 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:54:42,015 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:42,015 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:54:42,015 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:54:42,015 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:54:42,015 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. 2023-04-25 02:54:42,015 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. started... 2023-04-25 02:54:42,015 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. as already flushing 2023-04-25 02:54:42,015 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. 2023-04-25 02:54:42,015 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:54:42,015 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:54:42,016 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:42,016 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:54:42,016 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:54:42,018 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:42,018 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:42,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:42,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:42,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:42,018 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:42,018 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:54:42,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:42,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:42,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:42,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:42,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:42,020 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:54:42,020 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@56740a9f[Count = 0] remaining members to acquire global barrier 2023-04-25 02:54:42,020 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:54:42,020 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:42,021 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:42,021 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:42,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:42,022 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:54:42,022 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:54:42,022 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:54:42,022 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:42,022 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:54:42,023 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:42,023 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:54:42,023 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:42,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:42,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:42,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:42,023 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:54:42,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:42,023 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:42,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59214 deadline: 1682391342020, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:42,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:42,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:42,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:42,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:42,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:42,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:42,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:54:42,026 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:54:42,026 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:54:42,026 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:54:42,026 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:54:42,026 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:54:42,026 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:54:42,028 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:42,028 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:42,028 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:42,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:42,028 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:42,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:42,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:42,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:42,029 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:42,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:42,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:42,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:42,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:42,029 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:42,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:42,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:42,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:42,030 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:42,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:42,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:42,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:42,033 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:42,033 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:42,033 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:42,033 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:42,033 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:42,033 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:42,033 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:42,034 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:42,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:42,034 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:42,034 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:42,034 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:42,034 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:54:42,034 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:54:42,034 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:42,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:42,034 DEBUG [Thread-1512] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:54:42,034 DEBUG [Thread-1512] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:54:42,045 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:42,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59160 deadline: 1682391342043, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:42,046 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:42,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59186 deadline: 1682391342044, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:42,050 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:42,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59216 deadline: 1682391342047, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:42,134 DEBUG [Thread-1512] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:54:42,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:54:42,137 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:54:42,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:54:42,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:54:42,138 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:54:42,138 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:42,138 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:54:42,138 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:54:42,139 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:42,139 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:42,140 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:42,140 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:42,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:42,140 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:42,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:42,140 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:42,140 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:54:42,140 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:42,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:54:42,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:42,141 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:54:42,141 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:54:42,141 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:54:42,141 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:54:42,141 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:42,141 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:54:42,142 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:54:42,142 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:54:42,142 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. 2023-04-25 02:54:42,142 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. started... 2023-04-25 02:54:42,142 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. as already flushing 2023-04-25 02:54:42,142 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. 2023-04-25 02:54:42,142 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:54:42,142 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:54:42,142 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:42,142 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:54:42,142 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:54:42,143 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:42,143 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:42,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:42,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:42,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:42,144 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:42,144 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:54:42,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:42,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:42,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:42,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:42,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:42,145 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:54:42,145 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@384fdab[Count = 0] remaining members to acquire global barrier 2023-04-25 02:54:42,145 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:54:42,145 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:42,146 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:42,147 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:42,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:42,147 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:54:42,147 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:42,147 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:54:42,147 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:54:42,147 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:54:42,154 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:42,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:42,154 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:54:42,154 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:42,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:42,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:42,154 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:54:42,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:42,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:42,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:42,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:42,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:42,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:42,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:42,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:54:42,158 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:54:42,158 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:54:42,158 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:54:42,158 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:54:42,158 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:54:42,158 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:54:42,166 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:42,166 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:42,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:42,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:42,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:42,166 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:42,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:42,166 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:42,167 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:42,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:42,167 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:42,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:42,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:42,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:42,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:42,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:42,168 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:42,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:42,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:42,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:42,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:42,170 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:42,170 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:42,170 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:42,170 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:42,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:42,170 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:54:42,171 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:54:42,170 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:42,170 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:42,171 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:42,171 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:42,171 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:42,171 DEBUG [Thread-1512] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:54:42,171 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:54:42,171 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:42,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:42,171 DEBUG [Thread-1512] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:54:42,171 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:42,171 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:42,227 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:42,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59214 deadline: 1682391342226, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:42,247 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:42,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59160 deadline: 1682391342247, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:42,251 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:42,251 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59186 deadline: 1682391342248, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:42,253 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:42,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59216 deadline: 1682391342251, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:42,271 DEBUG [Thread-1512] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:54:42,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:54:42,273 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:54:42,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:54:42,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:54:42,274 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:54:42,274 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:42,274 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:54:42,274 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:54:42,275 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:42,275 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:42,276 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:42,276 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:42,276 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:42,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:42,277 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:42,277 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:54:42,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:42,277 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:42,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:54:42,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:42,277 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:54:42,278 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:54:42,278 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:54:42,278 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:54:42,278 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:42,278 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:54:42,278 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:54:42,278 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:54:42,278 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. 2023-04-25 02:54:42,278 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. started... 2023-04-25 02:54:42,278 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. as already flushing 2023-04-25 02:54:42,278 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. 2023-04-25 02:54:42,278 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:54:42,278 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:54:42,278 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:42,279 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:54:42,279 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:54:42,280 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:42,280 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:42,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:42,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:42,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:42,280 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:42,280 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:54:42,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:42,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:42,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:42,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:42,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:42,282 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:54:42,282 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2ee2db70[Count = 0] remaining members to acquire global barrier 2023-04-25 02:54:42,282 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:54:42,282 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:42,283 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:42,284 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:42,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:42,284 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:54:42,284 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:54:42,284 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:54:42,284 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:42,284 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:54:42,285 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:42,285 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:54:42,285 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:42,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:42,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:42,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:42,285 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:54:42,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:42,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:42,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:42,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:42,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:42,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:42,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:42,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:54:42,287 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:54:42,287 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:54:42,287 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:54:42,287 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:54:42,287 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:54:42,287 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:54:42,289 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:42,289 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:42,289 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:42,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:42,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:42,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:42,289 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:42,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:42,289 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:42,289 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:42,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:42,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:42,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:42,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:42,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:42,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:42,290 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:42,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:42,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:42,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:42,292 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=332 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp/B/b5855a635f154736af1bb491f09ecd2b 2023-04-25 02:54:42,295 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:42,295 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:42,295 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:42,295 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:42,295 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:42,295 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:42,295 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:42,295 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:42,295 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:42,295 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:42,295 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:42,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:42,295 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:54:42,295 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:54:42,295 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:42,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:42,295 DEBUG [Thread-1512] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:54:42,296 DEBUG [Thread-1512] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:54:42,296 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:54:42,396 DEBUG [Thread-1512] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:54:42,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:54:42,398 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:54:42,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:54:42,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:54:42,399 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:54:42,399 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:42,399 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:54:42,399 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:54:42,399 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:42,400 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:42,401 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:42,401 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:42,401 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:42,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:42,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:42,401 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:42,401 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:54:42,401 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:42,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:54:42,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:42,402 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:54:42,402 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:54:42,402 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:54:42,402 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:54:42,402 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:42,402 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:54:42,403 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:54:42,403 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:54:42,403 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. 2023-04-25 02:54:42,403 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. started... 2023-04-25 02:54:42,403 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. as already flushing 2023-04-25 02:54:42,403 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. 2023-04-25 02:54:42,403 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:54:42,403 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:54:42,403 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:42,403 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:54:42,403 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:54:42,404 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:42,404 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:42,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:42,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:42,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:42,405 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:42,405 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:54:42,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:42,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:42,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:42,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:42,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:42,406 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:54:42,406 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:54:42,406 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1b07c35[Count = 0] remaining members to acquire global barrier 2023-04-25 02:54:42,406 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:42,407 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:42,407 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:42,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:42,408 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:54:42,408 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:54:42,408 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:54:42,408 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:42,408 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:54:42,410 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:42,410 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:54:42,410 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:42,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:42,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:42,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:42,410 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:54:42,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:42,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:42,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:42,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:42,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:42,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:42,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:42,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:54:42,412 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:54:42,412 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:54:42,412 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:54:42,412 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:54:42,412 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:54:42,412 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:54:42,414 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:42,414 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:42,414 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:42,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:42,414 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:42,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:42,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:42,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:42,414 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:42,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:42,414 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:42,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:42,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:42,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:42,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:42,416 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:42,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:42,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:42,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:42,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:42,418 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:42,418 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:42,418 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:42,418 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:42,418 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:42,418 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:42,418 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:54:42,418 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:54:42,418 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:42,418 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:42,418 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:42,418 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:42,418 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:42,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:42,419 DEBUG [Thread-1512] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:54:42,419 DEBUG [Thread-1512] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:54:42,419 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:54:42,419 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:42,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:42,519 DEBUG [Thread-1512] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:54:42,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:54:42,521 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:54:42,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:54:42,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:54:42,522 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:54:42,522 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:42,522 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:54:42,522 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:54:42,523 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:42,523 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:42,525 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:42,525 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:42,525 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:42,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:42,525 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:42,525 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:54:42,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:42,525 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:42,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:54:42,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:42,526 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:54:42,526 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:54:42,526 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:54:42,526 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:54:42,526 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:42,526 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:54:42,527 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:54:42,527 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:54:42,527 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. 2023-04-25 02:54:42,527 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. started... 2023-04-25 02:54:42,527 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. as already flushing 2023-04-25 02:54:42,527 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. 2023-04-25 02:54:42,527 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:54:42,527 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:54:42,527 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:42,527 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:54:42,527 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:54:42,529 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:42,529 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:42,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:42,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:42,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:42,529 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:42,529 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:54:42,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:42,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:42,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:42,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:42,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:42,532 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:54:42,532 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@77e6be6e[Count = 0] remaining members to acquire global barrier 2023-04-25 02:54:42,532 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:54:42,532 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:42,532 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:42,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59214 deadline: 1682391342530, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:42,533 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:42,533 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:42,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:42,533 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:54:42,533 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:54:42,533 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:54:42,533 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:42,534 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:54:42,535 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:42,535 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:54:42,535 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:42,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:42,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:42,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:42,535 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:54:42,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:42,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:42,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:42,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:42,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:42,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:42,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:42,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:54:42,537 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:54:42,537 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:54:42,537 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:54:42,537 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:54:42,537 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:54:42,537 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:54:42,539 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:42,539 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:42,539 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:42,539 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:42,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:42,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:42,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:42,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:42,540 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:42,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:42,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:42,540 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:42,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:42,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:42,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:42,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:42,541 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:42,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:42,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:42,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:42,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:42,546 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:42,546 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:42,546 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:42,546 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:42,546 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:54:42,546 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:54:42,546 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:42,546 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:42,546 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:42,546 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:42,546 DEBUG [Thread-1512] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:54:42,546 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:42,546 DEBUG [Thread-1512] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:54:42,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:42,547 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:42,547 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:42,547 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:42,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:42,550 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:42,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59160 deadline: 1682391342549, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:42,557 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:42,557 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59186 deadline: 1682391342554, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:42,557 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:42,557 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59216 deadline: 1682391342555, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:42,647 DEBUG [Thread-1512] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:54:42,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:54:42,649 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:54:42,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:54:42,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:54:42,650 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:54:42,650 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:42,650 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:54:42,650 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:54:42,650 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:42,650 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:42,652 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:42,652 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:42,652 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:42,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:42,652 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:42,652 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:54:42,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:42,653 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:42,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:54:42,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:42,653 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:54:42,654 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:54:42,654 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:54:42,654 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:54:42,654 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:42,654 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:54:42,654 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:54:42,654 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:54:42,654 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. 2023-04-25 02:54:42,654 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. started... 2023-04-25 02:54:42,654 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. as already flushing 2023-04-25 02:54:42,654 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. 2023-04-25 02:54:42,654 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:54:42,654 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:54:42,654 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:42,654 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:54:42,654 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:54:42,657 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:42,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:42,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:42,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:42,657 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:42,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:42,657 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:42,657 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:54:42,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:42,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:42,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:42,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:42,658 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:54:42,658 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@53ab6c9c[Count = 0] remaining members to acquire global barrier 2023-04-25 02:54:42,658 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:54:42,659 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:42,660 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:42,660 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:42,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:42,660 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:54:42,660 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:54:42,660 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:54:42,660 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:42,660 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:54:42,661 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:42,661 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:54:42,661 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:42,661 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:54:42,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:42,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:42,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:42,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:42,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:42,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:42,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:42,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:42,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:42,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:42,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:54:42,664 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:54:42,664 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:54:42,664 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:54:42,664 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:54:42,664 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:54:42,664 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:54:42,665 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:42,665 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:42,665 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:42,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:42,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:42,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:42,665 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:42,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:42,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:42,666 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:42,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:42,666 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:42,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:42,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:42,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:42,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:42,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:42,667 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:42,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:42,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:42,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:42,671 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:42,671 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:42,671 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:42,671 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:42,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:42,671 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:42,671 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:42,671 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:54:42,671 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:54:42,671 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:42,671 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:42,672 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:42,672 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:42,672 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:42,672 DEBUG [Thread-1512] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:54:42,672 DEBUG [Thread-1512] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:54:42,672 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:42,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:42,704 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=332 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp/C/caf18a24d2864a2c9252a73580f5ee3e 2023-04-25 02:54:42,708 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp/A/a5d3cbd452244da5a1c55147119f7e0a as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A/a5d3cbd452244da5a1c55147119f7e0a 2023-04-25 02:54:42,712 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A/a5d3cbd452244da5a1c55147119f7e0a, entries=250, sequenceid=332, filesize=16.7 K 2023-04-25 02:54:42,713 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp/B/b5855a635f154736af1bb491f09ecd2b as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B/b5855a635f154736af1bb491f09ecd2b 2023-04-25 02:54:42,716 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B/b5855a635f154736af1bb491f09ecd2b, entries=150, sequenceid=332, filesize=11.9 K 2023-04-25 02:54:42,717 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp/C/caf18a24d2864a2c9252a73580f5ee3e as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C/caf18a24d2864a2c9252a73580f5ee3e 2023-04-25 02:54:42,721 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C/caf18a24d2864a2c9252a73580f5ee3e, entries=150, sequenceid=332, filesize=11.9 K 2023-04-25 02:54:42,722 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=129.71 KB/132820 for 40cf532a35b33bde962f4f28e36e887b in 856ms, sequenceid=332, compaction requested=true 2023-04-25 02:54:42,722 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 40cf532a35b33bde962f4f28e36e887b: 2023-04-25 02:54:42,722 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 02:54:42,722 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 02:54:42,722 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-25 02:54:42,722 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-25 02:54:42,722 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 02:54:42,724 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 74852 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-25 02:54:42,724 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HStore(1912): 40cf532a35b33bde962f4f28e36e887b/A is initiating minor compaction (all files) 2023-04-25 02:54:42,724 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 69972 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-25 02:54:42,724 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 40cf532a35b33bde962f4f28e36e887b/A in TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. 2023-04-25 02:54:42,724 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HStore(1912): 40cf532a35b33bde962f4f28e36e887b/B is initiating minor compaction (all files) 2023-04-25 02:54:42,724 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A/92c9b725031443e5b38491daade551ee, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A/eeace9228e5749bca7c0ad6499f5263e, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A/56fdb1660bae4bcd935540ba6f54b811, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A/0104577a6bdf452cae48db5ff23270f6, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A/3bec4c32bf214871996921c22931d36b, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A/a5d3cbd452244da5a1c55147119f7e0a] into tmpdir=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp, totalSize=73.1 K 2023-04-25 02:54:42,724 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 40cf532a35b33bde962f4f28e36e887b/B in TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. 2023-04-25 02:54:42,724 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B/370a415aaab24a6da9e885fe53ad6176, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B/f11aeab20c8b4454bc900c2fa1d8ad0c, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B/b373fbd5f50c43c1834fd6842d856498, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B/f5926eca392e48c6956392422ed54275, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B/cd0f47406b3e4b259e501639ad4f11c9, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B/b5855a635f154736af1bb491f09ecd2b] into tmpdir=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp, totalSize=68.3 K 2023-04-25 02:54:42,724 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting 92c9b725031443e5b38491daade551ee, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=12, earliestPutTs=1682391277705 2023-04-25 02:54:42,724 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting 370a415aaab24a6da9e885fe53ad6176, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=12, earliestPutTs=1682391277705 2023-04-25 02:54:42,725 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting eeace9228e5749bca7c0ad6499f5263e, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1682391277747 2023-04-25 02:54:42,725 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting f11aeab20c8b4454bc900c2fa1d8ad0c, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1682391277747 2023-04-25 02:54:42,725 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting 56fdb1660bae4bcd935540ba6f54b811, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=259, earliestPutTs=1682391278627 2023-04-25 02:54:42,725 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting b373fbd5f50c43c1834fd6842d856498, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=259, earliestPutTs=1682391278627 2023-04-25 02:54:42,728 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting 0104577a6bdf452cae48db5ff23270f6, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=267, earliestPutTs=1682391279742 2023-04-25 02:54:42,728 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting f5926eca392e48c6956392422ed54275, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=267, earliestPutTs=1682391279742 2023-04-25 02:54:42,728 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting 3bec4c32bf214871996921c22931d36b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=321, earliestPutTs=1682391280431 2023-04-25 02:54:42,729 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting cd0f47406b3e4b259e501639ad4f11c9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=321, earliestPutTs=1682391280431 2023-04-25 02:54:42,729 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting a5d3cbd452244da5a1c55147119f7e0a, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=332, earliestPutTs=1682391281554 2023-04-25 02:54:42,729 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting b5855a635f154736af1bb491f09ecd2b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=332, earliestPutTs=1682391281557 2023-04-25 02:54:42,750 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] throttle.PressureAwareThroughputController(145): 40cf532a35b33bde962f4f28e36e887b#A#compaction#201 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-25 02:54:42,759 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] throttle.PressureAwareThroughputController(145): 40cf532a35b33bde962f4f28e36e887b#B#compaction#202 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-25 02:54:42,767 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp/A/a9bb2db4c08e4bccbbd7ed61c73381ea as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A/a9bb2db4c08e4bccbbd7ed61c73381ea 2023-04-25 02:54:42,770 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(9158): Flush requested on 40cf532a35b33bde962f4f28e36e887b 2023-04-25 02:54:42,770 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 40cf532a35b33bde962f4f28e36e887b 3/3 column families, dataSize=136.42 KB heapSize=385.38 KB 2023-04-25 02:54:42,770 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 40cf532a35b33bde962f4f28e36e887b, store=A 2023-04-25 02:54:42,771 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 02:54:42,771 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 40cf532a35b33bde962f4f28e36e887b, store=B 2023-04-25 02:54:42,771 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 02:54:42,771 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 40cf532a35b33bde962f4f28e36e887b, store=C 2023-04-25 02:54:42,771 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 02:54:42,771 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 40cf532a35b33bde962f4f28e36e887b/A of 40cf532a35b33bde962f4f28e36e887b into a9bb2db4c08e4bccbbd7ed61c73381ea(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 02:54:42,771 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 40cf532a35b33bde962f4f28e36e887b: 2023-04-25 02:54:42,771 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b., storeName=40cf532a35b33bde962f4f28e36e887b/A, priority=10, startTime=1682391282722; duration=0sec 2023-04-25 02:54:42,771 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 02:54:42,772 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-25 02:54:42,772 DEBUG [Thread-1512] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:54:42,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:54:42,774 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 69972 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-25 02:54:42,774 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HStore(1912): 40cf532a35b33bde962f4f28e36e887b/C is initiating minor compaction (all files) 2023-04-25 02:54:42,774 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 40cf532a35b33bde962f4f28e36e887b/C in TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. 2023-04-25 02:54:42,774 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C/017f6cf1430b4dc5a4b4e605ead65115, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C/621545d991884cd6bda0d135d0fbfff9, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C/47f6457404874cc5943e7bcb2e8c22c1, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C/3f6bb3b42dc8411a848d42d32daecea6, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C/ab039aa53efc4464a6044bb43d22e721, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C/caf18a24d2864a2c9252a73580f5ee3e] into tmpdir=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp, totalSize=68.3 K 2023-04-25 02:54:42,774 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:54:42,774 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting 017f6cf1430b4dc5a4b4e605ead65115, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=12, earliestPutTs=1682391277705 2023-04-25 02:54:42,775 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting 621545d991884cd6bda0d135d0fbfff9, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1682391277747 2023-04-25 02:54:42,775 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting 47f6457404874cc5943e7bcb2e8c22c1, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=259, earliestPutTs=1682391278627 2023-04-25 02:54:42,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:54:42,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:54:42,776 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:54:42,776 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:42,776 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting 3f6bb3b42dc8411a848d42d32daecea6, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=267, earliestPutTs=1682391279742 2023-04-25 02:54:42,776 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:54:42,776 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:54:42,776 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting ab039aa53efc4464a6044bb43d22e721, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=321, earliestPutTs=1682391280431 2023-04-25 02:54:42,776 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:42,776 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:42,777 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting caf18a24d2864a2c9252a73580f5ee3e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=332, earliestPutTs=1682391281557 2023-04-25 02:54:42,778 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:42,778 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:42,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:42,778 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:42,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:42,778 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:42,778 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:54:42,779 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:42,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:54:42,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:42,779 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:54:42,780 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:54:42,780 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:54:42,780 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:54:42,780 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:42,780 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:54:42,788 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:54:42,788 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:54:42,788 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. 2023-04-25 02:54:42,788 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. started... 2023-04-25 02:54:42,788 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. as already flushing 2023-04-25 02:54:42,788 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. 2023-04-25 02:54:42,788 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:54:42,788 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:54:42,788 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:42,788 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:54:42,788 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:54:42,790 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:42,790 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:42,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:42,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:42,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:42,790 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:42,790 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:54:42,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:42,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:42,791 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] throttle.PressureAwareThroughputController(145): 40cf532a35b33bde962f4f28e36e887b#C#compaction#204 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-25 02:54:42,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:42,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:42,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:42,796 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:54:42,796 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6baa14c2[Count = 0] remaining members to acquire global barrier 2023-04-25 02:54:42,796 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:54:42,796 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:42,796 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=368 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp/A/41bcf169dead4de3b85c0b95ee2d8423 2023-04-25 02:54:42,797 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:42,797 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:42,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:42,798 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:54:42,798 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:54:42,798 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:54:42,799 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:42,799 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:54:42,800 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:42,800 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:54:42,800 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:42,800 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:54:42,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:42,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:42,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:42,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:42,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:42,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:42,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:42,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:42,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:42,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:42,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:54:42,805 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:54:42,805 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:54:42,805 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:54:42,805 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:54:42,805 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:54:42,805 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:54:42,807 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:42,807 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:42,807 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:42,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:42,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:42,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:42,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:42,807 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:42,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:42,808 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:42,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:42,808 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:42,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:42,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:42,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:42,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:42,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:42,809 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:42,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:42,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:42,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:42,811 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:42,811 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:42,811 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:42,811 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:54:42,811 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:54:42,811 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:42,811 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:42,811 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:42,811 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:42,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:42,811 DEBUG [Thread-1512] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:54:42,811 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:42,811 DEBUG [Thread-1512] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:54:42,811 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:42,811 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:42,811 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:42,812 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:42,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:42,812 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:42,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59158 deadline: 1682391342808, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:42,911 DEBUG [Thread-1512] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:54:42,912 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:54:42,913 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:54:42,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:54:42,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:54:42,915 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:54:42,915 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:42,915 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:54:42,915 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:54:42,915 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:42,915 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:42,916 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:42,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59158 deadline: 1682391342913, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:42,916 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:42,916 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:42,916 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:42,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:42,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:42,917 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:42,917 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:54:42,917 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:42,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:54:42,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:42,917 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:54:42,918 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:54:42,918 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:54:42,918 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:54:42,918 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:42,918 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:54:42,918 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:54:42,918 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:54:42,918 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. 2023-04-25 02:54:42,918 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. started... 2023-04-25 02:54:42,918 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. as already flushing 2023-04-25 02:54:42,918 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. 2023-04-25 02:54:42,918 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:54:42,918 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:54:42,918 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:42,919 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:54:42,919 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:54:42,921 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:42,921 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:42,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:42,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:42,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:42,922 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:42,922 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:54:42,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:42,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:42,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:42,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:42,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:42,923 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:54:42,923 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@69f98c56[Count = 0] remaining members to acquire global barrier 2023-04-25 02:54:42,923 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:54:42,923 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:42,924 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:42,924 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:42,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:42,924 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:54:42,924 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:54:42,924 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:54:42,924 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:42,925 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:54:42,926 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:42,926 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:54:42,926 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:42,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:42,926 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:54:42,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:42,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:42,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:42,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:42,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:42,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:42,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:42,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:42,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:42,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:54:42,928 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:54:42,928 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:54:42,928 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:54:42,928 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:54:42,928 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:54:42,928 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:54:42,929 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:42,929 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:42,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:42,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:42,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:42,930 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:42,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:42,930 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:42,930 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:42,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:42,930 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:42,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:42,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:42,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:42,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:42,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:42,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:42,932 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:42,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:42,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:42,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:42,937 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:42,937 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:42,937 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:42,937 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:42,937 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:54:42,937 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:54:42,937 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:42,937 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:42,938 DEBUG [Thread-1512] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:54:42,938 DEBUG [Thread-1512] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:54:42,937 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:42,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:42,937 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:42,938 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:42,938 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:42,938 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:42,938 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:42,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:43,035 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:43,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59214 deadline: 1682391343033, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:43,038 DEBUG [Thread-1512] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:54:43,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:54:43,040 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:54:43,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:54:43,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:54:43,041 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:54:43,041 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:43,041 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:54:43,041 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:54:43,041 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:43,041 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:43,043 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:43,043 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:43,043 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:43,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:43,043 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:43,043 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:54:43,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:43,043 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:43,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:54:43,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:43,044 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:54:43,044 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:54:43,044 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:54:43,044 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:43,044 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:54:43,044 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:54:43,044 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:54:43,044 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. 2023-04-25 02:54:43,044 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. started... 2023-04-25 02:54:43,045 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. as already flushing 2023-04-25 02:54:43,045 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. 2023-04-25 02:54:43,045 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:54:43,045 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:54:43,045 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:43,045 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:54:43,045 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:54:43,046 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:43,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:43,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:43,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:43,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:43,047 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:43,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:43,047 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:43,047 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:54:43,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:43,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:43,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:43,048 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:54:43,048 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2b2e4ac9[Count = 0] remaining members to acquire global barrier 2023-04-25 02:54:43,048 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:54:43,048 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:43,049 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:43,049 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:43,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:43,049 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:54:43,049 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:54:43,049 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:54:43,049 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:43,049 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:54:43,052 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:43,052 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:54:43,052 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:43,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:43,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:43,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:43,052 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:54:43,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:43,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:43,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:43,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:43,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:43,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:43,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:43,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:54:43,054 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:54:43,054 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:54:43,054 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:54:43,054 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:54:43,054 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:54:43,055 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:54:43,055 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:43,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59160 deadline: 1682391343053, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:43,057 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:43,057 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:43,057 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:43,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:43,057 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:43,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:43,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:43,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:43,057 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:43,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:43,058 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:43,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:43,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:43,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:43,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:43,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:43,059 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:43,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59186 deadline: 1682391343057, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:43,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:43,059 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:43,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:43,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:43,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:43,060 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:43,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59216 deadline: 1682391343058, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:43,061 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:43,061 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:43,061 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:43,061 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:43,061 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:43,061 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:43,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:43,061 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:54:43,062 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:54:43,061 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:43,062 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:43,062 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:43,062 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:43,062 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:43,062 DEBUG [Thread-1512] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:54:43,062 DEBUG [Thread-1512] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:54:43,062 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:43,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:43,122 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:43,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59158 deadline: 1682391343118, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:43,162 DEBUG [Thread-1512] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:54:43,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:54:43,165 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:54:43,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:54:43,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:54:43,166 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:54:43,166 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:43,166 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:54:43,166 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:54:43,167 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:43,167 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:43,168 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:43,168 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:43,168 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:54:43,169 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:43,169 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:43,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:43,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:43,169 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:43,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:54:43,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:43,170 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:54:43,170 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:54:43,170 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:54:43,171 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:54:43,171 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:43,171 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:54:43,171 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:54:43,171 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:54:43,171 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. 2023-04-25 02:54:43,172 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. started... 2023-04-25 02:54:43,172 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. as already flushing 2023-04-25 02:54:43,172 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. 2023-04-25 02:54:43,172 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:54:43,172 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:54:43,172 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:43,172 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:54:43,172 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:54:43,173 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp/B/9e9aa9ffa66f4e58b85b838149872849 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B/9e9aa9ffa66f4e58b85b838149872849 2023-04-25 02:54:43,174 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:43,174 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:43,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:43,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:43,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:43,175 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:43,175 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:54:43,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:43,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:43,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:43,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:43,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:43,177 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:54:43,177 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@23690cbc[Count = 0] remaining members to acquire global barrier 2023-04-25 02:54:43,177 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:54:43,177 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:43,178 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 40cf532a35b33bde962f4f28e36e887b/B of 40cf532a35b33bde962f4f28e36e887b into 9e9aa9ffa66f4e58b85b838149872849(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 02:54:43,178 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 40cf532a35b33bde962f4f28e36e887b: 2023-04-25 02:54:43,178 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b., storeName=40cf532a35b33bde962f4f28e36e887b/B, priority=10, startTime=1682391282722; duration=0sec 2023-04-25 02:54:43,178 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 02:54:43,178 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:43,178 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:43,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:43,178 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:54:43,178 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:54:43,178 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:54:43,178 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:43,179 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:54:43,180 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:43,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:43,180 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:54:43,180 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:43,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:43,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:43,180 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:54:43,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:43,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:43,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:43,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:43,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:43,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:43,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:43,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:54:43,183 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:54:43,183 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:54:43,183 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:54:43,183 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:54:43,183 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:54:43,183 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:54:43,184 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:43,184 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:43,184 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:43,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:43,184 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:43,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:43,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:43,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:43,185 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:43,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:43,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:43,185 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:43,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:43,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:43,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:43,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:43,186 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:43,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:43,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:43,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:43,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:43,190 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:43,190 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:43,190 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:43,190 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:43,190 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:54:43,190 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:54:43,190 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:43,190 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:54:43,190 DEBUG [Thread-1512] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:54:43,190 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:43,190 DEBUG [Thread-1512] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:54:43,190 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:43,190 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:43,190 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:43,190 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:43,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:43,190 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:43,191 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:43,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:43,207 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp/C/990b63ee975c4703b2e5ae29b7fe5c53 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C/990b63ee975c4703b2e5ae29b7fe5c53 2023-04-25 02:54:43,209 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=368 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp/B/395cb4f2860540d08f38f5626a6c12c5 2023-04-25 02:54:43,218 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 40cf532a35b33bde962f4f28e36e887b/C of 40cf532a35b33bde962f4f28e36e887b into 990b63ee975c4703b2e5ae29b7fe5c53(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 02:54:43,218 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 40cf532a35b33bde962f4f28e36e887b: 2023-04-25 02:54:43,218 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b., storeName=40cf532a35b33bde962f4f28e36e887b/C, priority=10, startTime=1682391282722; duration=0sec 2023-04-25 02:54:43,218 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 02:54:43,290 DEBUG [Thread-1512] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:54:43,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:54:43,293 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:54:43,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:54:43,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:54:43,294 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:54:43,294 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:43,294 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:54:43,294 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:54:43,294 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:43,294 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:43,295 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:43,295 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:43,295 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:43,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:43,296 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:43,296 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:54:43,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:43,296 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:43,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:54:43,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:43,296 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:54:43,297 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:54:43,297 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:54:43,297 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:54:43,297 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:43,297 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:54:43,297 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:54:43,297 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:54:43,298 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. 2023-04-25 02:54:43,298 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. started... 2023-04-25 02:54:43,298 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. as already flushing 2023-04-25 02:54:43,298 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. 2023-04-25 02:54:43,298 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:54:43,298 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:54:43,298 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:43,298 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:54:43,298 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:54:43,299 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:43,299 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:43,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:43,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:43,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:43,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:43,300 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:43,300 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:54:43,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:43,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:43,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:43,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:43,301 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:54:43,301 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:54:43,301 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@18de1017[Count = 0] remaining members to acquire global barrier 2023-04-25 02:54:43,301 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:43,302 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:43,302 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:43,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:43,302 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:54:43,302 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:54:43,302 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:54:43,303 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:43,303 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:54:43,305 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:43,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:43,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:43,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:43,305 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:54:43,305 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:43,305 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:54:43,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:43,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:43,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:43,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:43,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:43,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:43,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:43,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:54:43,307 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:54:43,307 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:54:43,307 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:54:43,307 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:54:43,307 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:54:43,307 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:54:43,308 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:43,308 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:43,309 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:43,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:43,309 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:43,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:43,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:43,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:43,309 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:43,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:43,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:43,309 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:43,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:43,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:43,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:43,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:43,310 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:43,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:43,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:43,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:43,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:43,313 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:43,313 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:43,313 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:43,313 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:43,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:43,313 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:54:43,313 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:54:43,313 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:43,313 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:43,313 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:43,313 DEBUG [Thread-1512] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:54:43,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:43,313 DEBUG [Thread-1512] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:54:43,313 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:43,314 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:43,314 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:43,314 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:43,314 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:43,414 DEBUG [Thread-1512] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:54:43,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:54:43,416 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:54:43,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:54:43,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:54:43,417 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:54:43,417 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:43,417 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:54:43,417 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:54:43,418 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:43,418 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:43,422 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:43,422 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:43,422 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:43,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:43,422 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:43,423 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:54:43,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:43,423 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:43,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:54:43,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:43,423 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:54:43,424 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:54:43,424 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:54:43,424 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:54:43,424 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:43,424 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:54:43,424 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:54:43,424 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:54:43,424 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. 2023-04-25 02:54:43,424 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. started... 2023-04-25 02:54:43,424 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. as already flushing 2023-04-25 02:54:43,424 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. 2023-04-25 02:54:43,424 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:54:43,424 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:54:43,424 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:43,424 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:54:43,424 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:54:43,426 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:43,426 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59158 deadline: 1682391343424, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:43,429 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:43,429 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:43,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:43,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:43,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:43,429 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:43,429 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:54:43,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:43,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:43,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:43,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:43,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:43,433 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:54:43,433 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3d60b466[Count = 0] remaining members to acquire global barrier 2023-04-25 02:54:43,433 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:54:43,433 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:43,434 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:43,434 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:43,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:43,434 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:54:43,434 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:54:43,434 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:54:43,434 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:43,434 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:54:43,435 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:43,435 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:54:43,435 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:43,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:43,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:43,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:43,435 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:54:43,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:43,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:43,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:43,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:43,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:43,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:43,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:43,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:54:43,438 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:54:43,438 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:54:43,438 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:54:43,438 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:54:43,438 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:54:43,438 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:54:43,441 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:43,441 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:43,441 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:43,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:43,441 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:43,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:43,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:43,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:43,441 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:43,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:43,441 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:43,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:43,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:43,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:43,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:43,442 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:43,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:43,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:43,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:43,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:43,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:43,445 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:43,445 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:43,445 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:43,445 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:43,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:43,445 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:43,445 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:54:43,445 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:43,445 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:43,445 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:43,445 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:43,445 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:43,445 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:54:43,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:43,445 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:43,445 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:43,445 DEBUG [Thread-1512] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:54:43,445 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:54:43,446 DEBUG [Thread-1512] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:54:43,546 DEBUG [Thread-1512] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:54:43,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:54:43,548 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:54:43,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:54:43,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:54:43,549 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:54:43,549 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:43,549 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:54:43,549 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:54:43,550 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:43,550 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:43,551 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:43,551 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:43,551 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:43,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:43,551 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:43,551 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:54:43,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:43,552 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:43,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:54:43,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:43,552 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:54:43,552 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:54:43,552 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:54:43,552 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:54:43,552 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:43,553 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:54:43,553 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:54:43,553 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:54:43,553 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. 2023-04-25 02:54:43,553 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. started... 2023-04-25 02:54:43,553 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. as already flushing 2023-04-25 02:54:43,553 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. 2023-04-25 02:54:43,553 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:54:43,553 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:54:43,553 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:43,553 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:54:43,553 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:54:43,555 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:43,555 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:43,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:43,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:43,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:43,555 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:43,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:43,555 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:54:43,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:43,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:43,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:43,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:43,557 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:54:43,557 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6441c901[Count = 0] remaining members to acquire global barrier 2023-04-25 02:54:43,557 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:54:43,557 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:43,560 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:43,560 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:43,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:43,560 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:54:43,561 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:54:43,561 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:54:43,561 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:43,561 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:54:43,562 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:54:43,562 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:43,562 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:43,562 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:54:43,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:43,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:43,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:43,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:43,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:43,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:43,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:43,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:43,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:43,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:43,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:54:43,564 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:54:43,564 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:54:43,564 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:54:43,564 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:54:43,564 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:54:43,564 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:54:43,566 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:43,566 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:43,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:43,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:43,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:43,566 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:43,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:43,566 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:43,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:43,567 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:43,567 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:43,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:43,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:43,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:43,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:43,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:43,568 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:43,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:43,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:43,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:43,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:43,569 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:43,569 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:43,569 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:43,569 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:43,570 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:43,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:43,570 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:43,570 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:43,570 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:43,570 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:43,570 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:54:43,570 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:43,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:43,570 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:43,570 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:54:43,570 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:43,570 DEBUG [Thread-1512] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:54:43,570 DEBUG [Thread-1512] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:54:43,570 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:54:43,627 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=368 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp/C/f5175a01513c4b729bc1ded565a8dc40 2023-04-25 02:54:43,632 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp/A/41bcf169dead4de3b85c0b95ee2d8423 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A/41bcf169dead4de3b85c0b95ee2d8423 2023-04-25 02:54:43,635 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A/41bcf169dead4de3b85c0b95ee2d8423, entries=200, sequenceid=368, filesize=14.3 K 2023-04-25 02:54:43,636 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp/B/395cb4f2860540d08f38f5626a6c12c5 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B/395cb4f2860540d08f38f5626a6c12c5 2023-04-25 02:54:43,640 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B/395cb4f2860540d08f38f5626a6c12c5, entries=150, sequenceid=368, filesize=11.9 K 2023-04-25 02:54:43,641 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp/C/f5175a01513c4b729bc1ded565a8dc40 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C/f5175a01513c4b729bc1ded565a8dc40 2023-04-25 02:54:43,645 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C/f5175a01513c4b729bc1ded565a8dc40, entries=150, sequenceid=368, filesize=11.9 K 2023-04-25 02:54:43,646 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~136.42 KB/139690, heapSize ~385.33 KB/394576, currentSize=46.96 KB/48090 for 40cf532a35b33bde962f4f28e36e887b in 876ms, sequenceid=368, compaction requested=false 2023-04-25 02:54:43,646 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 40cf532a35b33bde962f4f28e36e887b: 2023-04-25 02:54:43,671 DEBUG [Thread-1512] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:54:43,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:54:43,673 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:54:43,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:54:43,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:54:43,674 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:54:43,674 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:43,674 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:54:43,674 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:54:43,675 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:43,675 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:43,677 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:43,678 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:43,678 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:43,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:43,678 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:43,678 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:54:43,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:43,679 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:43,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:54:43,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:43,679 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:54:43,679 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:54:43,679 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:54:43,679 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:54:43,679 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:43,679 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:54:43,680 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:54:43,680 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:54:43,680 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. 2023-04-25 02:54:43,680 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. started... 2023-04-25 02:54:43,680 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 40cf532a35b33bde962f4f28e36e887b 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-25 02:54:43,680 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 40cf532a35b33bde962f4f28e36e887b, store=A 2023-04-25 02:54:43,680 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:54:43,680 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 40cf532a35b33bde962f4f28e36e887b, store=B 2023-04-25 02:54:43,680 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:54:43,681 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 40cf532a35b33bde962f4f28e36e887b, store=C 2023-04-25 02:54:43,681 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:54:43,698 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=381 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp/A/a0a64b540ec94a6b8d5b262e32b70b43 2023-04-25 02:54:43,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(9158): Flush requested on 40cf532a35b33bde962f4f28e36e887b 2023-04-25 02:54:43,932 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. as already flushing 2023-04-25 02:54:43,968 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682391283954, max timestamp=1682391283966Num uniques 859; 2023-04-25 02:54:43,968 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682391283954, max timestamp=1682391283966Num uniques 859; 2023-04-25 02:54:43,968 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-25 02:54:43,968 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-25 02:54:43,968 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682391283954, max timestamp=1682391283966Num uniques 859; 2023-04-25 02:54:43,968 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-25 02:54:44,019 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:44,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59158 deadline: 1682391344016, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:44,047 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:44,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59214 deadline: 1682391344044, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:44,064 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:44,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59186 deadline: 1682391344062, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:44,065 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:44,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59160 deadline: 1682391344063, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:44,067 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:44,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59216 deadline: 1682391344064, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:44,114 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=381 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp/B/10af95acf8654acaaf02994c65165195 2023-04-25 02:54:44,123 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:44,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59158 deadline: 1682391344120, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:44,326 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:44,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59158 deadline: 1682391344325, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:44,524 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=381 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp/C/accc96103a574b0d898f2420efedbf6c 2023-04-25 02:54:44,529 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp/A/a0a64b540ec94a6b8d5b262e32b70b43 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A/a0a64b540ec94a6b8d5b262e32b70b43 2023-04-25 02:54:44,533 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A/a0a64b540ec94a6b8d5b262e32b70b43, entries=150, sequenceid=381, filesize=11.9 K 2023-04-25 02:54:44,534 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp/B/10af95acf8654acaaf02994c65165195 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B/10af95acf8654acaaf02994c65165195 2023-04-25 02:54:44,539 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B/10af95acf8654acaaf02994c65165195, entries=150, sequenceid=381, filesize=11.9 K 2023-04-25 02:54:44,540 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp/C/accc96103a574b0d898f2420efedbf6c as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C/accc96103a574b0d898f2420efedbf6c 2023-04-25 02:54:44,543 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C/accc96103a574b0d898f2420efedbf6c, entries=150, sequenceid=381, filesize=11.9 K 2023-04-25 02:54:44,544 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 40cf532a35b33bde962f4f28e36e887b in 864ms, sequenceid=381, compaction requested=true 2023-04-25 02:54:44,544 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 40cf532a35b33bde962f4f28e36e887b: 2023-04-25 02:54:44,544 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. 2023-04-25 02:54:44,544 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:54:44,544 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:54:44,544 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:44,544 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:54:44,545 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:54:44,546 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:44,546 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:44,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:44,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:44,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:44,546 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:44,546 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:54:44,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:44,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:44,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:44,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:44,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:44,548 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:54:44,548 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@fa625d9[Count = 0] remaining members to acquire global barrier 2023-04-25 02:54:44,548 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:54:44,548 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:44,550 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:44,550 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:44,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:44,550 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:54:44,550 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:54:44,550 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:54:44,550 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:44,550 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:54:44,552 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:54:44,552 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:44,552 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:44,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:44,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:44,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:44,552 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:54:44,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:44,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:44,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:44,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:44,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:44,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:44,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:44,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:54:44,554 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:54:44,554 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:54:44,554 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:54:44,554 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:54:44,554 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:54:44,554 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:54:44,556 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:44,556 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:44,556 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:44,556 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:44,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:44,556 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:44,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:44,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:44,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:44,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:44,556 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:44,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:44,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:44,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:44,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:44,557 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:44,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:44,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:44,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:44,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:44,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:44,559 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:44,559 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:44,559 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:44,560 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:44,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:44,560 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:44,560 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:44,560 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:44,560 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:44,560 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:44,560 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:44,559 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:44,560 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:54:44,560 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:54:44,560 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:44,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:44,560 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:54:44,560 DEBUG [Thread-1512] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:54:44,560 DEBUG [Thread-1512] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:54:44,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(9158): Flush requested on 40cf532a35b33bde962f4f28e36e887b 2023-04-25 02:54:44,630 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 40cf532a35b33bde962f4f28e36e887b 3/3 column families, dataSize=140.89 KB heapSize=405.61 KB 2023-04-25 02:54:44,630 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 40cf532a35b33bde962f4f28e36e887b, store=A 2023-04-25 02:54:44,630 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-25 02:54:44,630 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 40cf532a35b33bde962f4f28e36e887b, store=B 2023-04-25 02:54:44,630 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-25 02:54:44,630 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 40cf532a35b33bde962f4f28e36e887b, store=C 2023-04-25 02:54:44,630 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-25 02:54:44,639 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=419 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp/A/ff82e765070b41c2844f1d44d154a8bb 2023-04-25 02:54:44,657 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:44,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59158 deadline: 1682391344653, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:44,660 DEBUG [Thread-1512] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:54:44,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:54:44,662 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:54:44,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:54:44,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:54:44,663 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:54:44,663 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:44,663 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:54:44,663 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:54:44,664 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:44,664 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:44,665 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:44,665 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:44,665 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:44,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:44,665 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:44,665 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:54:44,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:44,666 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:44,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:54:44,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:44,666 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:54:44,666 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:54:44,666 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:54:44,666 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:54:44,666 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:44,666 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:54:44,666 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:54:44,666 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:54:44,666 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. 2023-04-25 02:54:44,667 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. started... 2023-04-25 02:54:44,667 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. as already flushing 2023-04-25 02:54:44,667 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. 2023-04-25 02:54:44,667 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:54:44,667 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:54:44,667 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:44,667 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:54:44,667 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:54:44,669 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:44,669 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:44,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:44,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:44,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:44,670 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:44,670 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:54:44,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:44,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:44,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:44,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:44,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:44,671 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:54:44,671 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6710c292[Count = 0] remaining members to acquire global barrier 2023-04-25 02:54:44,671 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:54:44,671 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:44,672 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:44,672 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:44,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:44,673 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:54:44,673 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:54:44,673 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:54:44,673 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:44,673 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:54:44,674 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:54:44,674 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:44,674 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:44,674 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:54:44,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:44,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:44,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:44,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:44,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:44,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:44,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:44,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:44,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:44,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:44,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:54:44,676 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:54:44,676 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:54:44,676 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:54:44,676 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:54:44,676 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:54:44,676 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:54:44,678 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:44,678 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:44,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:44,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:44,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:44,678 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:44,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:44,678 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:44,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:44,679 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:44,679 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:44,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:44,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:44,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:44,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:44,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:44,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:44,680 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:44,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:44,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:44,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:44,682 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:44,682 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:44,682 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:44,682 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:44,682 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:44,682 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:44,682 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:44,682 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:54:44,682 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:54:44,682 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:54:44,682 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:44,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:44,683 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:44,683 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:44,683 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:44,683 DEBUG [Thread-1512] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:54:44,683 DEBUG [Thread-1512] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:54:44,683 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:44,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:44,760 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:44,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59158 deadline: 1682391344758, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:44,783 DEBUG [Thread-1512] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:54:44,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:54:44,785 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:54:44,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:54:44,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:54:44,787 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:54:44,787 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:44,787 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:54:44,787 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:54:44,787 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:44,787 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:44,788 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:44,789 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:44,789 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:44,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:44,789 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:44,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:44,789 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:54:44,789 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:44,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:54:44,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:44,790 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:54:44,790 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:54:44,790 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:54:44,790 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:54:44,790 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:44,790 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:54:44,790 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:54:44,790 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:54:44,790 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. 2023-04-25 02:54:44,790 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. started... 2023-04-25 02:54:44,790 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. as already flushing 2023-04-25 02:54:44,791 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. 2023-04-25 02:54:44,791 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:54:44,791 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:54:44,791 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:44,791 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:54:44,791 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:54:44,792 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:44,792 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:44,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:44,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:44,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:44,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:44,793 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:44,793 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:54:44,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:44,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:44,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:44,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:44,794 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:54:44,794 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:54:44,794 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:44,794 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5feb8992[Count = 0] remaining members to acquire global barrier 2023-04-25 02:54:44,796 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:44,796 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:44,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:44,796 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:54:44,796 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:54:44,796 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:54:44,796 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:44,797 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:54:44,798 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:54:44,798 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:44,798 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:44,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:44,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:44,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:44,798 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:54:44,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:44,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:44,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:44,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:44,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:44,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:44,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:44,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:54:44,800 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:54:44,800 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:54:44,800 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:54:44,800 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:54:44,800 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:54:44,800 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:54:44,802 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:44,802 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:44,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:44,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:44,802 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:44,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:44,802 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:44,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:44,802 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:44,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:44,802 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:44,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:44,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:44,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:44,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:44,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:44,804 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:44,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:44,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:44,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:44,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:44,806 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:44,806 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:44,806 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:44,806 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:44,806 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:54:44,806 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:54:44,806 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:44,806 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:44,806 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:44,806 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:54:44,806 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:44,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:44,807 DEBUG [Thread-1512] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:54:44,806 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:44,807 DEBUG [Thread-1512] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:54:44,807 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:44,807 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:44,807 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:44,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:44,907 DEBUG [Thread-1512] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:54:44,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:54:44,909 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:54:44,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:54:44,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:54:44,910 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:54:44,910 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:44,910 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:54:44,910 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:54:44,910 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:44,911 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:44,913 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:44,913 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:44,913 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:44,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:44,913 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:44,913 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:54:44,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:44,914 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:44,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:54:44,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:44,914 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:54:44,914 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:54:44,914 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:54:44,914 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:54:44,914 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:44,914 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:54:44,914 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:54:44,914 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:54:44,915 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. 2023-04-25 02:54:44,915 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. started... 2023-04-25 02:54:44,915 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. as already flushing 2023-04-25 02:54:44,915 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. 2023-04-25 02:54:44,915 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:54:44,915 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:54:44,915 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:44,915 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:54:44,915 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:54:44,916 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:44,916 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:44,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:44,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:44,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:44,917 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:44,917 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:54:44,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:44,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:44,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:44,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:44,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:44,918 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:54:44,918 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:54:44,918 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:44,918 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@35884aaf[Count = 0] remaining members to acquire global barrier 2023-04-25 02:54:44,919 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:44,919 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:44,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:44,919 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:54:44,919 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:54:44,919 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:54:44,919 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:44,919 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:54:44,920 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:44,920 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:54:44,920 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:44,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:44,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:44,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:44,920 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:54:44,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:44,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:44,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:44,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:44,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:44,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:44,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:44,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:54:44,923 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:54:44,923 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:54:44,923 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:54:44,923 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:54:44,923 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:54:44,923 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:54:44,925 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:44,925 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:44,926 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:44,926 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:44,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:44,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:44,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:44,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:44,926 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:44,926 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:44,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:44,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:44,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:44,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:44,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:44,927 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:44,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:44,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:44,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:44,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:44,930 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:44,930 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:44,930 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:44,930 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:44,930 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:44,930 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:54:44,930 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:54:44,930 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:44,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:44,930 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:44,930 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:54:44,930 DEBUG [Thread-1512] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:54:44,930 DEBUG [Thread-1512] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:54:44,930 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:44,930 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:44,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:44,930 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:44,930 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:44,930 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:44,963 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:44,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59158 deadline: 1682391344961, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:45,030 DEBUG [Thread-1512] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:54:45,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:54:45,032 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:54:45,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:54:45,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:54:45,034 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:54:45,034 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:45,034 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:54:45,034 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:54:45,034 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:45,034 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:45,035 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:45,035 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:45,035 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:45,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:45,036 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:45,036 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:54:45,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:45,036 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:45,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:54:45,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:45,037 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:54:45,037 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:54:45,037 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:54:45,037 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:54:45,037 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:45,037 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:54:45,037 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:54:45,037 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:54:45,037 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. 2023-04-25 02:54:45,037 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. started... 2023-04-25 02:54:45,037 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. as already flushing 2023-04-25 02:54:45,037 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. 2023-04-25 02:54:45,037 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:54:45,038 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:54:45,038 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:45,038 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:54:45,038 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:54:45,039 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:45,039 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:45,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:45,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:45,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:45,039 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:45,039 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:54:45,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:45,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:45,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:45,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:45,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:45,041 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:54:45,041 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@bbe292a[Count = 0] remaining members to acquire global barrier 2023-04-25 02:54:45,041 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:54:45,041 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:45,042 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:45,042 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:45,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:45,042 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:54:45,042 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:54:45,042 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:45,042 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:54:45,042 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:54:45,048 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:45,048 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:54:45,048 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:45,048 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:54:45,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:45,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:45,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:45,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:45,049 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=419 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp/B/3b3b650ac318479eae62b99d045f88aa 2023-04-25 02:54:45,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:45,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:45,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:45,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:45,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:45,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:45,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:54:45,051 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:54:45,051 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:54:45,051 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:54:45,051 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:54:45,051 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:54:45,051 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:54:45,053 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:45,053 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:45,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:45,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:45,053 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:45,053 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:45,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:45,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:45,054 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:45,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:45,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:45,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:45,054 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:45,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:45,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:45,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:45,055 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:45,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:45,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:45,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:45,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:45,059 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:45,059 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:45,059 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:45,059 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:45,059 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:54:45,059 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:45,059 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:45,059 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:45,059 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:45,059 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:54:45,059 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:54:45,059 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:45,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:45,059 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:45,059 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:45,060 DEBUG [Thread-1512] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:54:45,060 DEBUG [Thread-1512] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:54:45,060 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:45,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:45,160 DEBUG [Thread-1512] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:54:45,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:54:45,162 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:54:45,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:54:45,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:54:45,163 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:54:45,163 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:45,163 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:54:45,163 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:54:45,163 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:45,163 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:45,164 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:45,164 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:45,164 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:45,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:45,165 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:45,165 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:54:45,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:45,165 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:45,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:54:45,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:45,166 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:54:45,166 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:54:45,166 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:54:45,166 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:54:45,166 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:45,166 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:54:45,166 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:54:45,166 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:54:45,166 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. 2023-04-25 02:54:45,166 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. started... 2023-04-25 02:54:45,166 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. as already flushing 2023-04-25 02:54:45,167 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. 2023-04-25 02:54:45,167 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:54:45,167 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:54:45,167 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:45,167 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:54:45,167 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:54:45,172 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:45,172 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:45,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:45,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:45,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:45,172 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:45,172 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:54:45,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:45,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:45,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:45,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:45,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:45,174 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:54:45,174 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6a5b6e9d[Count = 0] remaining members to acquire global barrier 2023-04-25 02:54:45,174 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:54:45,174 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:45,175 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:45,175 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:45,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:45,175 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:54:45,175 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:45,175 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:54:45,175 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:54:45,175 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:54:45,176 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:54:45,176 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:45,176 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:45,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:45,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:45,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:45,176 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:54:45,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:45,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:45,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:45,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:45,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:45,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:45,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:45,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:54:45,178 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:54:45,178 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:54:45,178 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:54:45,179 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:54:45,179 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:54:45,179 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:54:45,180 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:45,180 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:45,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:45,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:45,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:45,180 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:45,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:45,180 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:45,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:45,180 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:45,180 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:45,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:45,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:45,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:45,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:45,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:45,182 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:45,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:45,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:45,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:45,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:45,184 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:45,184 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:45,184 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:45,184 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:45,184 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:54:45,184 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:54:45,184 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:45,184 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:45,184 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:45,185 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:45,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:45,185 DEBUG [Thread-1512] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:54:45,185 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:45,185 DEBUG [Thread-1512] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:54:45,185 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:45,185 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:45,185 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:45,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:45,268 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:45,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59158 deadline: 1682391345264, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:45,285 DEBUG [Thread-1512] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:54:45,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:54:45,287 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:54:45,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:54:45,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:54:45,288 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:54:45,288 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:45,288 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:54:45,288 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:54:45,289 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:45,289 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:45,290 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:45,290 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:45,290 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:45,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:45,290 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:45,290 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:54:45,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:45,291 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:45,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:54:45,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:45,291 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:54:45,291 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:54:45,291 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:54:45,291 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:54:45,291 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:45,291 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:54:45,292 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:54:45,292 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. 2023-04-25 02:54:45,292 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:54:45,292 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. started... 2023-04-25 02:54:45,292 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. as already flushing 2023-04-25 02:54:45,292 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. 2023-04-25 02:54:45,292 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:54:45,292 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:54:45,292 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:45,292 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:54:45,292 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:54:45,293 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:45,293 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:45,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:45,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:45,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:45,294 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:45,294 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:54:45,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:45,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:45,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:45,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:45,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:45,295 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:54:45,295 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@69ffdd5d[Count = 0] remaining members to acquire global barrier 2023-04-25 02:54:45,295 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:54:45,295 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:45,296 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:45,296 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:45,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:45,297 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:54:45,297 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:54:45,297 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:54:45,297 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:45,297 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:54:45,298 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:45,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:45,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:45,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:45,299 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:54:45,299 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:45,299 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:54:45,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:45,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:45,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:45,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:45,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:45,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:45,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:45,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:54:45,301 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:54:45,301 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:54:45,302 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:54:45,302 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:54:45,302 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:54:45,302 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:54:45,303 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:45,303 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:45,303 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:45,303 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:45,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:45,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:45,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:45,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:45,303 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:45,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:45,304 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:45,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:45,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:45,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:45,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:45,305 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:45,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:45,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:45,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:45,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:45,307 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:45,307 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:45,307 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:45,307 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:45,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:45,307 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:45,307 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:45,307 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:54:45,307 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:54:45,307 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:45,307 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:45,307 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:45,307 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:45,307 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:45,307 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:54:45,307 DEBUG [Thread-1512] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:54:45,308 DEBUG [Thread-1512] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:54:45,308 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:45,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:45,408 DEBUG [Thread-1512] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:54:45,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:54:45,410 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:54:45,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:54:45,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:54:45,411 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:54:45,411 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:45,411 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:54:45,411 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:54:45,412 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:45,412 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:45,413 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:45,413 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:45,413 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:45,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:45,413 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:45,413 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:54:45,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:45,414 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:45,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:54:45,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:45,414 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:54:45,414 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:54:45,414 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:54:45,414 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:54:45,414 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:45,415 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:54:45,415 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:54:45,415 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:54:45,415 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. 2023-04-25 02:54:45,415 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. started... 2023-04-25 02:54:45,415 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. as already flushing 2023-04-25 02:54:45,415 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. 2023-04-25 02:54:45,415 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:54:45,415 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:54:45,415 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:45,415 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:54:45,415 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:54:45,421 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:45,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:45,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:45,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:45,422 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:45,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:45,422 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:45,422 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:54:45,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:45,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:45,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:45,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:45,423 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:54:45,423 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1e553fff[Count = 0] remaining members to acquire global barrier 2023-04-25 02:54:45,423 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:54:45,423 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:45,424 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:45,424 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:45,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:45,424 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:54:45,424 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:54:45,424 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:54:45,424 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:45,425 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:54:45,425 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:54:45,425 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:45,425 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:45,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:45,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:45,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:45,425 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:54:45,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:45,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:45,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:45,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:45,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:45,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:45,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:45,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:54:45,428 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:54:45,428 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:54:45,428 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:54:45,428 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:54:45,428 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:54:45,428 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:54:45,429 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:45,429 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:45,429 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:45,429 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:45,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:45,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:45,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:45,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:45,430 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:45,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:45,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:45,430 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:45,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:45,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:45,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:45,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:45,431 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:45,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:45,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:45,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:45,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:45,437 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:45,437 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:45,437 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:45,437 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:45,437 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:45,437 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:54:45,437 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:54:45,437 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:45,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:45,437 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:54:45,437 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:45,437 DEBUG [Thread-1512] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:54:45,437 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:45,437 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:45,438 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:45,437 DEBUG [Thread-1512] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:54:45,438 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:45,438 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:45,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:45,461 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=419 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp/C/851700bd208e4350b1eb976dc306e1a5 2023-04-25 02:54:45,465 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp/A/ff82e765070b41c2844f1d44d154a8bb as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A/ff82e765070b41c2844f1d44d154a8bb 2023-04-25 02:54:45,469 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A/ff82e765070b41c2844f1d44d154a8bb, entries=150, sequenceid=419, filesize=11.9 K 2023-04-25 02:54:45,470 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp/B/3b3b650ac318479eae62b99d045f88aa as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B/3b3b650ac318479eae62b99d045f88aa 2023-04-25 02:54:45,474 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B/3b3b650ac318479eae62b99d045f88aa, entries=150, sequenceid=419, filesize=11.9 K 2023-04-25 02:54:45,475 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp/C/851700bd208e4350b1eb976dc306e1a5 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C/851700bd208e4350b1eb976dc306e1a5 2023-04-25 02:54:45,478 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C/851700bd208e4350b1eb976dc306e1a5, entries=150, sequenceid=419, filesize=11.9 K 2023-04-25 02:54:45,479 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~405.56 KB/415296, currentSize=40.25 KB/41220 for 40cf532a35b33bde962f4f28e36e887b in 850ms, sequenceid=419, compaction requested=true 2023-04-25 02:54:45,479 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 40cf532a35b33bde962f4f28e36e887b: 2023-04-25 02:54:45,479 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 02:54:45,479 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-25 02:54:45,479 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-25 02:54:45,479 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 02:54:45,480 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 02:54:45,481 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 51390 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-25 02:54:45,481 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HStore(1912): 40cf532a35b33bde962f4f28e36e887b/A is initiating minor compaction (all files) 2023-04-25 02:54:45,481 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 40cf532a35b33bde962f4f28e36e887b/A in TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. 2023-04-25 02:54:45,481 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A/a9bb2db4c08e4bccbbd7ed61c73381ea, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A/41bcf169dead4de3b85c0b95ee2d8423, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A/a0a64b540ec94a6b8d5b262e32b70b43, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A/ff82e765070b41c2844f1d44d154a8bb] into tmpdir=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp, totalSize=50.2 K 2023-04-25 02:54:45,482 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48950 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-25 02:54:45,482 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HStore(1912): 40cf532a35b33bde962f4f28e36e887b/B is initiating minor compaction (all files) 2023-04-25 02:54:45,482 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 40cf532a35b33bde962f4f28e36e887b/B in TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. 2023-04-25 02:54:45,482 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B/9e9aa9ffa66f4e58b85b838149872849, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B/395cb4f2860540d08f38f5626a6c12c5, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B/10af95acf8654acaaf02994c65165195, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B/3b3b650ac318479eae62b99d045f88aa] into tmpdir=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp, totalSize=47.8 K 2023-04-25 02:54:45,482 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting a9bb2db4c08e4bccbbd7ed61c73381ea, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=332, earliestPutTs=1682391281557 2023-04-25 02:54:45,483 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting 9e9aa9ffa66f4e58b85b838149872849, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=332, earliestPutTs=1682391281557 2023-04-25 02:54:45,483 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting 41bcf169dead4de3b85c0b95ee2d8423, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=368, earliestPutTs=1682391281932 2023-04-25 02:54:45,483 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting 395cb4f2860540d08f38f5626a6c12c5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=368, earliestPutTs=1682391281932 2023-04-25 02:54:45,483 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting a0a64b540ec94a6b8d5b262e32b70b43, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=381, earliestPutTs=1682391282783 2023-04-25 02:54:45,483 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting 10af95acf8654acaaf02994c65165195, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=381, earliestPutTs=1682391282783 2023-04-25 02:54:45,484 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting ff82e765070b41c2844f1d44d154a8bb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=419, earliestPutTs=1682391283983 2023-04-25 02:54:45,484 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting 3b3b650ac318479eae62b99d045f88aa, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=419, earliestPutTs=1682391283983 2023-04-25 02:54:45,497 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] throttle.PressureAwareThroughputController(145): 40cf532a35b33bde962f4f28e36e887b#A#compaction#213 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-25 02:54:45,506 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] throttle.PressureAwareThroughputController(145): 40cf532a35b33bde962f4f28e36e887b#B#compaction#214 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-25 02:54:45,538 DEBUG [Thread-1512] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:54:45,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:54:45,539 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:54:45,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:54:45,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:54:45,541 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:54:45,541 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:45,541 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:54:45,541 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:54:45,541 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:45,541 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:45,542 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:45,542 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:45,542 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:45,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:45,542 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:45,542 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:54:45,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:45,543 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:45,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:54:45,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:45,543 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:54:45,543 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:54:45,543 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:54:45,543 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:54:45,544 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:45,544 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:54:45,544 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:54:45,544 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:54:45,544 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. 2023-04-25 02:54:45,544 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. started... 2023-04-25 02:54:45,544 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 40cf532a35b33bde962f4f28e36e887b 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-04-25 02:54:45,544 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 40cf532a35b33bde962f4f28e36e887b, store=A 2023-04-25 02:54:45,544 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:54:45,544 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 40cf532a35b33bde962f4f28e36e887b, store=B 2023-04-25 02:54:45,544 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:54:45,544 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 40cf532a35b33bde962f4f28e36e887b, store=C 2023-04-25 02:54:45,544 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:54:45,567 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=428 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp/A/3204165bcff647c1a5dc2becf3becf8d 2023-04-25 02:54:45,583 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=428 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp/B/76482a48794d46a3ab6ff78f6a7ab9da 2023-04-25 02:54:45,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(9158): Flush requested on 40cf532a35b33bde962f4f28e36e887b 2023-04-25 02:54:45,773 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. as already flushing 2023-04-25 02:54:45,828 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682391285808, max timestamp=1682391285826Num uniques 859; 2023-04-25 02:54:45,828 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-25 02:54:45,828 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682391285808, max timestamp=1682391285826Num uniques 859; 2023-04-25 02:54:45,828 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-25 02:54:45,828 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682391285808, max timestamp=1682391285826Num uniques 859; 2023-04-25 02:54:45,828 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-25 02:54:45,884 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1682391285854, max timestamp=1682391285881Num uniques 1018; 2023-04-25 02:54:45,884 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-25 02:54:45,884 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1682391285854, max timestamp=1682391285881Num uniques 1018; 2023-04-25 02:54:45,884 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-25 02:54:45,884 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1682391285854, max timestamp=1682391285881Num uniques 1018; 2023-04-25 02:54:45,884 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-25 02:54:45,912 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp/A/c4734762cf6644aeab94a7ebd511bb1a as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A/c4734762cf6644aeab94a7ebd511bb1a 2023-04-25 02:54:45,916 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp/B/9d666fbce5414c1eb8a974973d4ee483 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B/9d666fbce5414c1eb8a974973d4ee483 2023-04-25 02:54:45,921 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 40cf532a35b33bde962f4f28e36e887b/A of 40cf532a35b33bde962f4f28e36e887b into c4734762cf6644aeab94a7ebd511bb1a(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 02:54:45,921 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 40cf532a35b33bde962f4f28e36e887b: 2023-04-25 02:54:45,921 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b., storeName=40cf532a35b33bde962f4f28e36e887b/A, priority=12, startTime=1682391285479; duration=0sec 2023-04-25 02:54:45,921 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 02:54:45,921 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-25 02:54:45,922 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48950 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-25 02:54:45,923 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HStore(1912): 40cf532a35b33bde962f4f28e36e887b/C is initiating minor compaction (all files) 2023-04-25 02:54:45,923 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 40cf532a35b33bde962f4f28e36e887b/C in TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. 2023-04-25 02:54:45,923 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C/990b63ee975c4703b2e5ae29b7fe5c53, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C/f5175a01513c4b729bc1ded565a8dc40, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C/accc96103a574b0d898f2420efedbf6c, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C/851700bd208e4350b1eb976dc306e1a5] into tmpdir=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp, totalSize=47.8 K 2023-04-25 02:54:45,923 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting 990b63ee975c4703b2e5ae29b7fe5c53, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=332, earliestPutTs=1682391281557 2023-04-25 02:54:45,924 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting f5175a01513c4b729bc1ded565a8dc40, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=368, earliestPutTs=1682391281932 2023-04-25 02:54:45,924 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting accc96103a574b0d898f2420efedbf6c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=381, earliestPutTs=1682391282783 2023-04-25 02:54:45,924 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting 851700bd208e4350b1eb976dc306e1a5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=419, earliestPutTs=1682391283983 2023-04-25 02:54:45,932 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] throttle.PressureAwareThroughputController(145): 40cf532a35b33bde962f4f28e36e887b#C#compaction#218 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-25 02:54:45,933 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 40cf532a35b33bde962f4f28e36e887b/B of 40cf532a35b33bde962f4f28e36e887b into 9d666fbce5414c1eb8a974973d4ee483(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 02:54:45,933 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 40cf532a35b33bde962f4f28e36e887b: 2023-04-25 02:54:45,933 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b., storeName=40cf532a35b33bde962f4f28e36e887b/B, priority=12, startTime=1682391285479; duration=0sec 2023-04-25 02:54:45,933 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 02:54:45,945 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1682391285927, max timestamp=1682391285943Num uniques 1027; 2023-04-25 02:54:45,945 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1682391285927, max timestamp=1682391285943Num uniques 1027; 2023-04-25 02:54:45,946 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-04-25 02:54:45,946 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-04-25 02:54:45,946 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1682391285927, max timestamp=1682391285943Num uniques 1027; 2023-04-25 02:54:45,946 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-04-25 02:54:46,015 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=428 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp/C/20d9ec3f2ab341608e7313227e1ffbf8 2023-04-25 02:54:46,020 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp/A/3204165bcff647c1a5dc2becf3becf8d as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A/3204165bcff647c1a5dc2becf3becf8d 2023-04-25 02:54:46,022 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1682391286004, max timestamp=1682391286020Num uniques 1036; 2023-04-25 02:54:46,022 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=8516, suffix heap size=135712,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=186 2023-04-25 02:54:46,023 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1682391286004, max timestamp=1682391286020Num uniques 1036; 2023-04-25 02:54:46,023 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1682391286004, max timestamp=1682391286020Num uniques 1036; 2023-04-25 02:54:46,023 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=8516, suffix heap size=135712,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=186 2023-04-25 02:54:46,023 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=8516, suffix heap size=135712,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=186 2023-04-25 02:54:46,026 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A/3204165bcff647c1a5dc2becf3becf8d, entries=100, sequenceid=428, filesize=9.5 K 2023-04-25 02:54:46,027 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp/B/76482a48794d46a3ab6ff78f6a7ab9da as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B/76482a48794d46a3ab6ff78f6a7ab9da 2023-04-25 02:54:46,031 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B/76482a48794d46a3ab6ff78f6a7ab9da, entries=100, sequenceid=428, filesize=9.5 K 2023-04-25 02:54:46,032 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp/C/20d9ec3f2ab341608e7313227e1ffbf8 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C/20d9ec3f2ab341608e7313227e1ffbf8 2023-04-25 02:54:46,037 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C/20d9ec3f2ab341608e7313227e1ffbf8, entries=100, sequenceid=428, filesize=9.5 K 2023-04-25 02:54:46,047 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=46.96 KB/48090 for 40cf532a35b33bde962f4f28e36e887b in 503ms, sequenceid=428, compaction requested=false 2023-04-25 02:54:46,047 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 40cf532a35b33bde962f4f28e36e887b: 2023-04-25 02:54:46,047 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. 2023-04-25 02:54:46,047 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:54:46,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(9158): Flush requested on 40cf532a35b33bde962f4f28e36e887b 2023-04-25 02:54:46,047 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:54:46,048 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:46,047 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 40cf532a35b33bde962f4f28e36e887b 3/3 column families, dataSize=53.67 KB heapSize=143.39 KB 2023-04-25 02:54:46,048 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:54:46,048 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:54:46,048 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 40cf532a35b33bde962f4f28e36e887b, store=A 2023-04-25 02:54:46,048 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 02:54:46,048 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 40cf532a35b33bde962f4f28e36e887b, store=B 2023-04-25 02:54:46,048 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 02:54:46,048 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 40cf532a35b33bde962f4f28e36e887b, store=C 2023-04-25 02:54:46,048 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 02:54:46,049 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:46,049 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:46,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:46,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:46,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:46,049 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:46,049 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:54:46,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:46,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:46,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:46,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:46,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:46,056 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:54:46,056 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@64d8b266[Count = 0] remaining members to acquire global barrier 2023-04-25 02:54:46,056 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:54:46,056 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:46,057 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:46,057 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:46,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:46,057 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:54:46,057 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:54:46,058 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:54:46,058 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:46,058 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:54:46,064 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:54:46,064 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=506 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp/A/b004de39cde34321ad418bb4c33804e6 2023-04-25 02:54:46,064 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:46,064 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:46,064 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:54:46,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:46,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:46,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:46,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:46,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:46,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:46,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:46,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:46,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:46,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:46,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:54:46,070 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:54:46,070 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:54:46,070 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:54:46,070 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:54:46,070 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:54:46,070 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:54:46,072 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:46,072 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:46,072 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:46,073 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:46,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:46,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:46,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:46,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:46,073 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:46,073 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:46,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:46,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:46,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:46,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:46,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:46,075 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:46,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:46,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:46,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:46,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:46,077 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:46,077 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:46,077 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:46,077 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:46,077 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:54:46,077 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:54:46,077 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:46,077 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:46,077 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:46,077 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:46,077 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:46,077 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:46,077 DEBUG [Thread-1512] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:54:46,077 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:46,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:46,077 DEBUG [Thread-1512] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:54:46,078 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:54:46,078 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:46,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:46,087 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682391286077, max timestamp=1682391286085Num uniques 859; 2023-04-25 02:54:46,087 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682391286082, max timestamp=1682391286085Num uniques 859; 2023-04-25 02:54:46,087 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-25 02:54:46,087 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-25 02:54:46,087 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682391286082, max timestamp=1682391286085Num uniques 859; 2023-04-25 02:54:46,087 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-25 02:54:46,103 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:46,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59216 deadline: 1682391346099, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:46,105 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:46,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59214 deadline: 1682391346100, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:46,106 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:46,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59160 deadline: 1682391346103, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:46,106 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:46,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 218 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59158 deadline: 1682391346103, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:46,106 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:46,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59186 deadline: 1682391346104, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:46,177 DEBUG [Thread-1512] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:54:46,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:54:46,179 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:54:46,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:54:46,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:54:46,181 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:54:46,181 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:46,181 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:54:46,181 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:54:46,181 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:46,181 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:46,182 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:46,182 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:46,182 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:46,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:46,183 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:46,183 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:54:46,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:46,183 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:46,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:54:46,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:46,183 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:54:46,184 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:54:46,184 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:54:46,184 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:54:46,184 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:46,184 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:54:46,184 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:54:46,184 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:54:46,184 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. 2023-04-25 02:54:46,184 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. started... 2023-04-25 02:54:46,184 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. as already flushing 2023-04-25 02:54:46,184 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. 2023-04-25 02:54:46,184 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:54:46,184 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:54:46,184 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:46,184 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:54:46,184 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:54:46,187 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:46,187 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:46,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:46,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:46,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:46,187 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:46,187 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:54:46,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:46,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:46,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:46,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:46,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:46,188 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:54:46,188 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@554617e6[Count = 0] remaining members to acquire global barrier 2023-04-25 02:54:46,188 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:54:46,189 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:46,190 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:46,190 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:46,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:46,190 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:54:46,190 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:54:46,190 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:54:46,190 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:46,190 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:54:46,191 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:46,191 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:54:46,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:46,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:46,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:46,191 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:46,191 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:54:46,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:46,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:46,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:46,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:46,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:46,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:46,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:46,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:54:46,193 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:54:46,193 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:54:46,193 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:54:46,193 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:54:46,194 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:54:46,194 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:54:46,195 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:46,195 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:46,195 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:46,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:46,195 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:46,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:46,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:46,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:46,195 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:46,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:46,195 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:46,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:46,196 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:46,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:46,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:46,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:46,199 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:46,199 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:46,199 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:46,199 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:46,199 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:46,199 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:46,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:46,199 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:46,199 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:46,199 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:54:46,199 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:54:46,199 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:46,199 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:46,199 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:46,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:46,199 DEBUG [Thread-1512] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:54:46,200 DEBUG [Thread-1512] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:54:46,200 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:46,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:46,200 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:54:46,206 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:46,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59216 deadline: 1682391346204, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:46,207 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:46,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59214 deadline: 1682391346206, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:46,209 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:46,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59160 deadline: 1682391346207, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:46,209 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:46,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 220 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59158 deadline: 1682391346207, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:46,210 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:46,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59186 deadline: 1682391346207, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:46,300 DEBUG [Thread-1512] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:54:46,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:54:46,302 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:54:46,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:54:46,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:54:46,303 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:54:46,303 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:46,303 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:54:46,303 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:54:46,304 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:46,304 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:46,305 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:46,305 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:46,305 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:46,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:46,305 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:46,305 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:54:46,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:46,305 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:46,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:54:46,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:46,306 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:54:46,306 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:54:46,306 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:54:46,306 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:54:46,306 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:46,306 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:54:46,306 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:54:46,306 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:54:46,306 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. 2023-04-25 02:54:46,307 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. started... 2023-04-25 02:54:46,307 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. as already flushing 2023-04-25 02:54:46,307 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. 2023-04-25 02:54:46,307 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:54:46,307 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:54:46,307 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:46,307 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:54:46,307 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:54:46,308 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:46,308 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:46,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:46,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:46,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:46,309 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:46,309 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:54:46,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:46,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:46,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:46,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:46,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:46,310 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:54:46,310 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@42c42d26[Count = 0] remaining members to acquire global barrier 2023-04-25 02:54:46,310 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:54:46,310 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:46,311 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:46,311 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:46,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:46,311 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:54:46,311 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:54:46,311 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:54:46,311 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:46,311 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:54:46,313 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:46,313 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:54:46,313 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:46,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:46,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:46,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:46,313 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:54:46,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:46,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:46,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:46,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:46,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:46,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:46,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:46,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:54:46,315 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:54:46,316 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:54:46,316 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:54:46,316 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:54:46,316 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:54:46,316 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:54:46,317 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:46,317 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:46,317 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:46,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:46,317 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:46,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:46,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:46,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:46,317 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:46,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:46,318 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:46,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:46,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:46,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:46,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:46,319 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:46,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:46,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:46,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:46,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:46,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:46,324 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:46,324 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:46,324 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:46,324 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:46,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:46,324 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:46,324 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:46,324 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:54:46,325 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:54:46,325 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:46,325 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:46,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:46,325 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:54:46,325 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:46,325 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:46,325 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:46,325 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:46,325 DEBUG [Thread-1512] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:54:46,325 DEBUG [Thread-1512] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:54:46,341 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp/C/45ec236adc5e40049ce316032a7a3e21 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C/45ec236adc5e40049ce316032a7a3e21 2023-04-25 02:54:46,345 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 40cf532a35b33bde962f4f28e36e887b/C of 40cf532a35b33bde962f4f28e36e887b into 45ec236adc5e40049ce316032a7a3e21(size=12.2 K), total size for store is 21.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 02:54:46,345 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 40cf532a35b33bde962f4f28e36e887b: 2023-04-25 02:54:46,345 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b., storeName=40cf532a35b33bde962f4f28e36e887b/C, priority=12, startTime=1682391285480; duration=0sec 2023-04-25 02:54:46,345 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 02:54:46,409 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:46,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59216 deadline: 1682391346407, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:46,409 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:46,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59214 deadline: 1682391346408, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:46,412 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:46,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 222 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59158 deadline: 1682391346410, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:46,413 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:46,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59186 deadline: 1682391346411, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:46,413 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:46,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59160 deadline: 1682391346411, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:46,425 DEBUG [Thread-1512] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:54:46,426 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:54:46,427 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:54:46,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:54:46,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:54:46,429 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:54:46,429 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:46,429 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:54:46,429 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:54:46,429 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:46,429 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:46,432 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:46,432 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:46,432 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:46,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:46,432 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:46,432 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:54:46,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:46,432 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:46,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:54:46,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:46,433 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:54:46,433 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:54:46,433 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:54:46,433 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:54:46,433 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:46,433 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:54:46,434 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:54:46,434 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:54:46,434 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. 2023-04-25 02:54:46,434 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. started... 2023-04-25 02:54:46,434 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. as already flushing 2023-04-25 02:54:46,434 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. 2023-04-25 02:54:46,434 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:54:46,434 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:54:46,434 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:46,434 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:54:46,434 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:54:46,436 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:46,436 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:46,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:46,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:46,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:46,436 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:46,436 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:54:46,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:46,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:46,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:46,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:46,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:46,438 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:54:46,438 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@75d94846[Count = 0] remaining members to acquire global barrier 2023-04-25 02:54:46,438 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:54:46,438 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:46,439 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:46,439 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:46,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:46,439 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:54:46,439 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:46,439 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:54:46,439 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:54:46,439 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:54:46,440 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:46,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:46,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:46,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:46,440 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:54:46,441 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:46,441 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:54:46,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:46,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:46,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:46,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:46,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:46,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:46,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:46,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:54:46,443 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:54:46,443 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:54:46,443 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:54:46,443 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:54:46,443 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:54:46,443 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:54:46,445 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:46,446 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:46,446 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:46,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:46,446 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:46,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:46,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:46,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:46,446 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:46,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:46,447 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:46,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:46,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:46,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:46,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:46,448 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:46,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:46,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:46,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:46,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:46,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:46,450 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:46,450 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:46,450 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:46,450 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:54:46,450 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:54:46,450 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:46,450 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:46,450 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:46,450 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:46,450 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:46,450 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:46,450 DEBUG [Thread-1512] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:54:46,450 DEBUG [Thread-1512] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:54:46,450 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:46,450 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:54:46,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:46,450 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:46,451 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:46,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:46,477 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=506 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp/B/8975a22ef3e24d10bc69c68356164b57 2023-04-25 02:54:46,492 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=506 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp/C/08a16b8aa41a4f5da673c932b5a8b41c 2023-04-25 02:54:46,496 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp/A/b004de39cde34321ad418bb4c33804e6 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A/b004de39cde34321ad418bb4c33804e6 2023-04-25 02:54:46,500 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A/b004de39cde34321ad418bb4c33804e6, entries=200, sequenceid=506, filesize=14.3 K 2023-04-25 02:54:46,501 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp/B/8975a22ef3e24d10bc69c68356164b57 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B/8975a22ef3e24d10bc69c68356164b57 2023-04-25 02:54:46,507 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B/8975a22ef3e24d10bc69c68356164b57, entries=150, sequenceid=506, filesize=11.9 K 2023-04-25 02:54:46,508 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp/C/08a16b8aa41a4f5da673c932b5a8b41c as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C/08a16b8aa41a4f5da673c932b5a8b41c 2023-04-25 02:54:46,518 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C/08a16b8aa41a4f5da673c932b5a8b41c, entries=150, sequenceid=506, filesize=11.9 K 2023-04-25 02:54:46,519 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~143.34 KB/146784, currentSize=138.65 KB/141980 for 40cf532a35b33bde962f4f28e36e887b in 472ms, sequenceid=506, compaction requested=true 2023-04-25 02:54:46,519 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 40cf532a35b33bde962f4f28e36e887b: 2023-04-25 02:54:46,519 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 02:54:46,519 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-25 02:54:46,519 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-25 02:54:46,519 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 02:54:46,520 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 02:54:46,524 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36896 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-25 02:54:46,526 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34456 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-25 02:54:46,526 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HStore(1912): 40cf532a35b33bde962f4f28e36e887b/A is initiating minor compaction (all files) 2023-04-25 02:54:46,526 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HStore(1912): 40cf532a35b33bde962f4f28e36e887b/B is initiating minor compaction (all files) 2023-04-25 02:54:46,526 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 40cf532a35b33bde962f4f28e36e887b/A in TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. 2023-04-25 02:54:46,526 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 40cf532a35b33bde962f4f28e36e887b/B in TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. 2023-04-25 02:54:46,526 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A/c4734762cf6644aeab94a7ebd511bb1a, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A/3204165bcff647c1a5dc2becf3becf8d, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A/b004de39cde34321ad418bb4c33804e6] into tmpdir=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp, totalSize=36.0 K 2023-04-25 02:54:46,526 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B/9d666fbce5414c1eb8a974973d4ee483, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B/76482a48794d46a3ab6ff78f6a7ab9da, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B/8975a22ef3e24d10bc69c68356164b57] into tmpdir=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp, totalSize=33.6 K 2023-04-25 02:54:46,526 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting c4734762cf6644aeab94a7ebd511bb1a, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=419, earliestPutTs=1682391283983 2023-04-25 02:54:46,527 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting 9d666fbce5414c1eb8a974973d4ee483, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=419, earliestPutTs=1682391283983 2023-04-25 02:54:46,527 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting 3204165bcff647c1a5dc2becf3becf8d, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=428, earliestPutTs=1682391284643 2023-04-25 02:54:46,527 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting 76482a48794d46a3ab6ff78f6a7ab9da, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=428, earliestPutTs=1682391284643 2023-04-25 02:54:46,528 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting b004de39cde34321ad418bb4c33804e6, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=506, earliestPutTs=1682391286004 2023-04-25 02:54:46,528 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting 8975a22ef3e24d10bc69c68356164b57, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=506, earliestPutTs=1682391286004 2023-04-25 02:54:46,541 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] throttle.PressureAwareThroughputController(145): 40cf532a35b33bde962f4f28e36e887b#B#compaction#222 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-25 02:54:46,549 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] throttle.PressureAwareThroughputController(145): 40cf532a35b33bde962f4f28e36e887b#A#compaction#223 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-25 02:54:46,550 DEBUG [Thread-1512] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:54:46,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:54:46,553 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:54:46,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:54:46,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:54:46,572 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:54:46,572 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:46,572 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:54:46,572 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:54:46,572 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:46,573 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:46,576 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:46,576 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:46,576 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:46,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:46,577 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:46,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:46,577 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:54:46,577 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:46,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:54:46,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:46,580 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:54:46,580 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:54:46,580 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:54:46,580 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:54:46,580 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:46,580 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:54:46,581 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:54:46,581 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:54:46,581 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. 2023-04-25 02:54:46,581 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. started... 2023-04-25 02:54:46,581 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 40cf532a35b33bde962f4f28e36e887b 3/3 column families, dataSize=138.65 KB heapSize=395.14 KB 2023-04-25 02:54:46,581 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 40cf532a35b33bde962f4f28e36e887b, store=A 2023-04-25 02:54:46,581 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 02:54:46,581 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 40cf532a35b33bde962f4f28e36e887b, store=B 2023-04-25 02:54:46,582 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 02:54:46,582 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 40cf532a35b33bde962f4f28e36e887b, store=C 2023-04-25 02:54:46,582 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 02:54:46,586 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp/A/cbe8e6cba8e04939a8da94367ec0e635 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A/cbe8e6cba8e04939a8da94367ec0e635 2023-04-25 02:54:46,591 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 40cf532a35b33bde962f4f28e36e887b/A of 40cf532a35b33bde962f4f28e36e887b into cbe8e6cba8e04939a8da94367ec0e635(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 02:54:46,591 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 40cf532a35b33bde962f4f28e36e887b: 2023-04-25 02:54:46,591 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b., storeName=40cf532a35b33bde962f4f28e36e887b/A, priority=13, startTime=1682391286519; duration=0sec 2023-04-25 02:54:46,591 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 02:54:46,591 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-25 02:54:46,592 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34456 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-25 02:54:46,592 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HStore(1912): 40cf532a35b33bde962f4f28e36e887b/C is initiating minor compaction (all files) 2023-04-25 02:54:46,592 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 40cf532a35b33bde962f4f28e36e887b/C in TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. 2023-04-25 02:54:46,592 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C/45ec236adc5e40049ce316032a7a3e21, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C/20d9ec3f2ab341608e7313227e1ffbf8, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C/08a16b8aa41a4f5da673c932b5a8b41c] into tmpdir=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp, totalSize=33.6 K 2023-04-25 02:54:46,593 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting 45ec236adc5e40049ce316032a7a3e21, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=419, earliestPutTs=1682391283983 2023-04-25 02:54:46,593 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting 20d9ec3f2ab341608e7313227e1ffbf8, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=428, earliestPutTs=1682391284643 2023-04-25 02:54:46,594 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting 08a16b8aa41a4f5da673c932b5a8b41c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=506, earliestPutTs=1682391286004 2023-04-25 02:54:46,595 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=544 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp/A/f80956a04fd44e29953e6f8db595c477 2023-04-25 02:54:46,600 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] throttle.PressureAwareThroughputController(145): 40cf532a35b33bde962f4f28e36e887b#C#compaction#225 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-25 02:54:46,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(9158): Flush requested on 40cf532a35b33bde962f4f28e36e887b 2023-04-25 02:54:46,714 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. as already flushing 2023-04-25 02:54:46,719 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:46,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59160 deadline: 1682391346715, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:46,720 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:46,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59186 deadline: 1682391346718, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:46,721 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:46,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 225 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59158 deadline: 1682391346718, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:46,721 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:46,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59216 deadline: 1682391346719, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:46,721 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:46,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59214 deadline: 1682391346720, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:46,824 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:46,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59186 deadline: 1682391346821, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:46,824 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:46,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59158 deadline: 1682391346822, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:46,824 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:46,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59216 deadline: 1682391346822, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:46,824 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:46,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59214 deadline: 1682391346822, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:46,985 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp/B/47cbd8623b5b4bf6a7b97e67c55451fd as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B/47cbd8623b5b4bf6a7b97e67c55451fd 2023-04-25 02:54:46,991 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 40cf532a35b33bde962f4f28e36e887b/B of 40cf532a35b33bde962f4f28e36e887b into 47cbd8623b5b4bf6a7b97e67c55451fd(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 02:54:46,991 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 40cf532a35b33bde962f4f28e36e887b: 2023-04-25 02:54:46,991 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b., storeName=40cf532a35b33bde962f4f28e36e887b/B, priority=13, startTime=1682391286519; duration=0sec 2023-04-25 02:54:46,991 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 02:54:47,015 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp/C/7ee5c141bc454854a59b483fc5a6028d as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C/7ee5c141bc454854a59b483fc5a6028d 2023-04-25 02:54:47,018 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=544 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp/B/6c05444a679b4b46a4b38a7569e2cf74 2023-04-25 02:54:47,020 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 40cf532a35b33bde962f4f28e36e887b/C of 40cf532a35b33bde962f4f28e36e887b into 7ee5c141bc454854a59b483fc5a6028d(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 02:54:47,020 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 40cf532a35b33bde962f4f28e36e887b: 2023-04-25 02:54:47,020 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b., storeName=40cf532a35b33bde962f4f28e36e887b/C, priority=13, startTime=1682391286520; duration=0sec 2023-04-25 02:54:47,020 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 02:54:47,028 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:47,028 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59186 deadline: 1682391347025, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:47,028 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:47,028 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 229 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59158 deadline: 1682391347025, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:47,028 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:47,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59214 deadline: 1682391347025, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:47,029 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:47,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59216 deadline: 1682391347026, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:47,034 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=544 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp/C/aa98107974264ee8b9ef7be771d8e5e5 2023-04-25 02:54:47,038 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp/A/f80956a04fd44e29953e6f8db595c477 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A/f80956a04fd44e29953e6f8db595c477 2023-04-25 02:54:47,042 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A/f80956a04fd44e29953e6f8db595c477, entries=150, sequenceid=544, filesize=11.9 K 2023-04-25 02:54:47,043 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp/B/6c05444a679b4b46a4b38a7569e2cf74 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B/6c05444a679b4b46a4b38a7569e2cf74 2023-04-25 02:54:47,048 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B/6c05444a679b4b46a4b38a7569e2cf74, entries=150, sequenceid=544, filesize=11.9 K 2023-04-25 02:54:47,049 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp/C/aa98107974264ee8b9ef7be771d8e5e5 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C/aa98107974264ee8b9ef7be771d8e5e5 2023-04-25 02:54:47,053 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C/aa98107974264ee8b9ef7be771d8e5e5, entries=150, sequenceid=544, filesize=11.9 K 2023-04-25 02:54:47,054 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~138.65 KB/141980, heapSize ~395.09 KB/404576, currentSize=40.25 KB/41220 for 40cf532a35b33bde962f4f28e36e887b in 472ms, sequenceid=544, compaction requested=false 2023-04-25 02:54:47,054 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 40cf532a35b33bde962f4f28e36e887b: 2023-04-25 02:54:47,054 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. 2023-04-25 02:54:47,054 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:54:47,054 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:54:47,054 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:47,054 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:54:47,054 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:54:47,055 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:47,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:47,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:47,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:47,056 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:47,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:47,056 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:47,056 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:54:47,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:47,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:47,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:47,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:47,057 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:54:47,057 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@54339f19[Count = 0] remaining members to acquire global barrier 2023-04-25 02:54:47,057 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:54:47,057 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:47,060 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:47,060 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:47,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:47,060 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:54:47,060 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:54:47,060 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:47,060 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:54:47,060 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:54:47,061 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:54:47,061 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:47,061 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:47,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:47,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:47,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:47,062 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:54:47,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:47,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:47,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:47,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:47,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:47,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:47,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:47,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:54:47,065 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:54:47,065 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:54:47,065 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:54:47,065 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:54:47,065 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:54:47,065 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:54:47,066 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:47,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:47,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:47,067 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:47,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:47,067 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:47,067 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:47,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:47,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:47,067 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:47,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:47,067 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:47,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:47,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:47,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:47,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:47,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:47,069 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:47,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:47,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:47,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:47,071 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:47,071 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:47,071 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:47,071 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:47,071 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:47,071 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:47,071 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:54:47,071 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:47,071 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:47,071 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:47,071 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:47,071 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:54:47,071 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:54:47,071 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:47,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:47,072 DEBUG [Thread-1512] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:54:47,072 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:47,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:47,072 DEBUG [Thread-1512] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:54:47,172 DEBUG [Thread-1512] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:54:47,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:54:47,174 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:54:47,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:54:47,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:54:47,175 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:54:47,176 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:47,176 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:54:47,176 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:54:47,176 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:47,176 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:47,178 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:47,178 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:47,179 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:47,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:47,179 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:47,179 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:54:47,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:47,179 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:47,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:54:47,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:47,180 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:54:47,180 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:54:47,180 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:54:47,180 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:54:47,180 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:47,180 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:54:47,180 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:54:47,180 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:54:47,181 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. 2023-04-25 02:54:47,181 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. started... 2023-04-25 02:54:47,181 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 40cf532a35b33bde962f4f28e36e887b 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-04-25 02:54:47,181 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 40cf532a35b33bde962f4f28e36e887b, store=A 2023-04-25 02:54:47,181 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:54:47,181 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 40cf532a35b33bde962f4f28e36e887b, store=B 2023-04-25 02:54:47,181 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:54:47,181 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 40cf532a35b33bde962f4f28e36e887b, store=C 2023-04-25 02:54:47,181 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:54:47,188 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=556 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp/A/818b8e53d66a4bc1b70ccda74e5a67a9 2023-04-25 02:54:47,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(9158): Flush requested on 40cf532a35b33bde962f4f28e36e887b 2023-04-25 02:54:47,225 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. as already flushing 2023-04-25 02:54:47,269 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682391287255, max timestamp=1682391287267Num uniques 859; 2023-04-25 02:54:47,269 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682391287255, max timestamp=1682391287267Num uniques 859; 2023-04-25 02:54:47,269 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-25 02:54:47,269 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682391287255, max timestamp=1682391287267Num uniques 859; 2023-04-25 02:54:47,269 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-25 02:54:47,269 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-25 02:54:47,318 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1682391287295, max timestamp=1682391287316Num uniques 1018; 2023-04-25 02:54:47,318 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-25 02:54:47,319 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1682391287295, max timestamp=1682391287316Num uniques 1018; 2023-04-25 02:54:47,319 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1682391287295, max timestamp=1682391287316Num uniques 1018; 2023-04-25 02:54:47,319 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-25 02:54:47,319 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-25 02:54:47,346 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:47,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59160 deadline: 1682391347341, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:47,346 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1682391287339, max timestamp=1682391287345Num uniques 1027; 2023-04-25 02:54:47,346 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-04-25 02:54:47,347 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1682391287339, max timestamp=1682391287345Num uniques 1027; 2023-04-25 02:54:47,347 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-04-25 02:54:47,347 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8100, totalHeapSize=18648, min timestamp=1682391287339, max timestamp=1682391287345Num uniques 1027; 2023-04-25 02:54:47,347 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=47030, new segment data size=8100, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-04-25 02:54:47,367 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:47,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59214 deadline: 1682391347364, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:47,368 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1682391287361, max timestamp=1682391287366Num uniques 1036; 2023-04-25 02:54:47,368 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=8516, suffix heap size=135712,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=186 2023-04-25 02:54:47,369 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1682391287361, max timestamp=1682391287366Num uniques 1036; 2023-04-25 02:54:47,369 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=8516, suffix heap size=135712,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=186 2023-04-25 02:54:47,369 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1682391287361, max timestamp=1682391287366Num uniques 1036; 2023-04-25 02:54:47,369 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=8516, suffix heap size=135712,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=186 2023-04-25 02:54:47,393 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:47,393 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 243 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59158 deadline: 1682391347389, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:47,394 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1682391287369, max timestamp=1682391287392Num uniques 1045; 2023-04-25 02:54:47,394 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=11220, suffix heap size=136648,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=245 2023-04-25 02:54:47,394 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1682391287369, max timestamp=1682391287392Num uniques 1045; 2023-04-25 02:54:47,394 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=11220, suffix heap size=136648,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=245 2023-04-25 02:54:47,394 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1682391287369, max timestamp=1682391287392Num uniques 1045; 2023-04-25 02:54:47,394 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=11220, suffix heap size=136648,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=245 2023-04-25 02:54:47,438 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:47,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59186 deadline: 1682391347435, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:47,439 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:47,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59216 deadline: 1682391347437, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:47,450 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:47,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59160 deadline: 1682391347448, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:47,469 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:47,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59214 deadline: 1682391347468, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:47,496 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:47,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 245 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59158 deadline: 1682391347494, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:47,540 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:47,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59186 deadline: 1682391347539, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:47,542 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:47,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59216 deadline: 1682391347540, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:47,599 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=556 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp/B/e45a857578e9461382e39d5313efd5df 2023-04-25 02:54:47,653 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:47,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59160 deadline: 1682391347651, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:47,671 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:47,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59214 deadline: 1682391347670, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:47,698 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:47,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 247 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59158 deadline: 1682391347698, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:47,744 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:47,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59186 deadline: 1682391347743, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:47,744 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:47,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59216 deadline: 1682391347743, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:47,957 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:47,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59160 deadline: 1682391347955, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:47,975 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:47,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59214 deadline: 1682391347974, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:48,003 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:48,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 249 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59158 deadline: 1682391348002, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:48,009 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=556 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp/C/c8591ba6d2734211b1962f3142544a13 2023-04-25 02:54:48,013 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp/A/818b8e53d66a4bc1b70ccda74e5a67a9 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A/818b8e53d66a4bc1b70ccda74e5a67a9 2023-04-25 02:54:48,017 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A/818b8e53d66a4bc1b70ccda74e5a67a9, entries=150, sequenceid=556, filesize=11.9 K 2023-04-25 02:54:48,018 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp/B/e45a857578e9461382e39d5313efd5df as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B/e45a857578e9461382e39d5313efd5df 2023-04-25 02:54:48,022 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B/e45a857578e9461382e39d5313efd5df, entries=150, sequenceid=556, filesize=11.9 K 2023-04-25 02:54:48,022 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp/C/c8591ba6d2734211b1962f3142544a13 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C/c8591ba6d2734211b1962f3142544a13 2023-04-25 02:54:48,026 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C/c8591ba6d2734211b1962f3142544a13, entries=150, sequenceid=556, filesize=11.9 K 2023-04-25 02:54:48,027 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=140.89 KB/144270 for 40cf532a35b33bde962f4f28e36e887b in 846ms, sequenceid=556, compaction requested=true 2023-04-25 02:54:48,027 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 40cf532a35b33bde962f4f28e36e887b: 2023-04-25 02:54:48,027 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. 2023-04-25 02:54:48,027 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:54:48,027 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:54:48,027 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:48,027 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:54:48,027 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:54:48,028 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:48,028 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:48,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:48,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:48,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:48,029 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:48,029 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:54:48,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:48,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:48,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:48,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:48,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:48,030 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:54:48,030 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@312f4b5e[Count = 0] remaining members to acquire global barrier 2023-04-25 02:54:48,030 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:54:48,030 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:48,031 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:48,031 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:48,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:48,031 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:54:48,031 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:54:48,031 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:54:48,031 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:48,031 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:54:48,033 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:54:48,033 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:48,033 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:48,033 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:54:48,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:48,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:48,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:48,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:48,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:48,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:48,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:48,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:48,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:48,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:48,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:54:48,035 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:54:48,035 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:54:48,035 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:54:48,035 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:54:48,035 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:54:48,035 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:54:48,038 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:48,038 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:48,038 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:48,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:48,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:48,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:48,038 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:48,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:48,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:48,039 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:48,039 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:48,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:48,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:48,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:48,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:48,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:48,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:48,040 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:48,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:48,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:48,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:48,042 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:48,042 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:48,042 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:48,042 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:48,042 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:48,042 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:54:48,042 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:54:48,042 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:48,043 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:48,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:48,042 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:48,043 DEBUG [Thread-1512] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:54:48,043 DEBUG [Thread-1512] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:54:48,042 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:48,043 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:48,043 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:48,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:48,043 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:48,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(9158): Flush requested on 40cf532a35b33bde962f4f28e36e887b 2023-04-25 02:54:48,048 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 40cf532a35b33bde962f4f28e36e887b 3/3 column families, dataSize=147.60 KB heapSize=416.77 KB 2023-04-25 02:54:48,049 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 40cf532a35b33bde962f4f28e36e887b, store=A 2023-04-25 02:54:48,049 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-25 02:54:48,049 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 40cf532a35b33bde962f4f28e36e887b, store=B 2023-04-25 02:54:48,049 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-25 02:54:48,049 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 40cf532a35b33bde962f4f28e36e887b, store=C 2023-04-25 02:54:48,049 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-25 02:54:48,058 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=663 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp/A/9dfeafb5844843d2a4f4d1193640fc58 2023-04-25 02:54:48,060 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:48,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59186 deadline: 1682391348056, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:48,060 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:48,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59216 deadline: 1682391348056, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:48,143 DEBUG [Thread-1512] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:54:48,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:54:48,145 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:54:48,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:54:48,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:54:48,146 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:54:48,146 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:48,146 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:54:48,146 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:54:48,147 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:48,147 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:48,150 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:48,150 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:48,150 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:48,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:48,150 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:48,150 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:54:48,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:48,151 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:48,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:54:48,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:48,151 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:54:48,151 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:54:48,151 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:54:48,151 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:54:48,151 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:48,151 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:54:48,152 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:54:48,152 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:54:48,152 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. 2023-04-25 02:54:48,152 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. started... 2023-04-25 02:54:48,152 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. as already flushing 2023-04-25 02:54:48,152 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. 2023-04-25 02:54:48,152 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:54:48,152 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:54:48,152 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:48,152 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:54:48,152 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:54:48,154 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:48,154 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:48,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:48,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:48,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:48,155 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:48,155 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:54:48,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:48,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:48,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:48,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:48,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:48,156 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:54:48,156 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@52e1ed40[Count = 0] remaining members to acquire global barrier 2023-04-25 02:54:48,156 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:54:48,156 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:48,157 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:48,157 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:48,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:48,157 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:54:48,157 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:54:48,157 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:54:48,157 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:48,158 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:54:48,159 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:54:48,159 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:48,159 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:48,159 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:54:48,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:48,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:48,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:48,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:48,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:48,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:48,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:48,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:48,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:48,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:48,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:54:48,162 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:54:48,162 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:54:48,162 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:54:48,162 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:54:48,162 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:54:48,162 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:54:48,163 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:48,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59186 deadline: 1682391348161, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:48,163 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:48,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59216 deadline: 1682391348161, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:48,163 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:48,163 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:48,163 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:48,163 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:48,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:48,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:48,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:48,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:48,164 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:48,164 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:48,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:48,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:48,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:48,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:48,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:48,165 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:48,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:48,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:48,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:48,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:48,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:48,168 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:48,168 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:48,168 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:48,168 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:48,168 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:48,168 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:54:48,168 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:54:48,168 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:48,168 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:48,169 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:54:48,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:48,169 DEBUG [Thread-1512] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:54:48,169 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:48,169 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:48,169 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:48,169 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:48,169 DEBUG [Thread-1512] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:54:48,169 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:48,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:48,269 DEBUG [Thread-1512] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:54:48,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:54:48,271 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:54:48,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:54:48,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:54:48,273 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:54:48,273 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:48,273 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:54:48,273 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:54:48,274 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:48,274 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:48,275 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:48,275 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:48,275 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:48,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:48,275 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:48,275 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:54:48,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:48,276 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:48,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:54:48,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:48,276 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:54:48,276 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:54:48,276 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:54:48,276 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:54:48,276 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:48,276 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:54:48,277 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:54:48,277 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:54:48,277 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. 2023-04-25 02:54:48,277 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. started... 2023-04-25 02:54:48,277 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. as already flushing 2023-04-25 02:54:48,277 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. 2023-04-25 02:54:48,277 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:54:48,277 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:54:48,277 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:48,277 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:54:48,277 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:54:48,278 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:48,278 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:48,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:48,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:48,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:48,278 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:48,279 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:54:48,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:48,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:48,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:48,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:48,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:48,280 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:54:48,280 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6d451422[Count = 0] remaining members to acquire global barrier 2023-04-25 02:54:48,280 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:54:48,280 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:48,281 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:48,281 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:48,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:48,281 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:54:48,281 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:54:48,281 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:54:48,281 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:48,281 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:54:48,283 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:54:48,283 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:48,283 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:48,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:48,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:48,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:48,283 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:54:48,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:48,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:48,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:48,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:48,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:48,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:48,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:48,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:54:48,286 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:54:48,286 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:54:48,286 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:54:48,286 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:54:48,286 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:54:48,286 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:54:48,288 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:48,288 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:48,288 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:48,288 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:48,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:48,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:48,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:48,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:48,288 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:48,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:48,288 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:48,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:48,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:48,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:48,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:48,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:48,289 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:48,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:48,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:48,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:48,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:48,291 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:48,291 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:48,291 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:48,291 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:54:48,292 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:54:48,291 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:48,291 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:48,292 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:48,292 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:48,292 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:48,292 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:48,291 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:48,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:48,291 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:54:48,291 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:48,292 DEBUG [Thread-1512] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:54:48,292 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:48,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:48,292 DEBUG [Thread-1512] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:54:48,366 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:48,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59186 deadline: 1682391348364, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:48,367 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:48,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59216 deadline: 1682391348364, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:48,392 DEBUG [Thread-1512] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:54:48,393 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:54:48,394 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:54:48,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:54:48,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:54:48,395 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:54:48,395 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:48,396 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:54:48,396 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:54:48,396 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:48,396 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:48,402 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:48,402 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:48,402 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:48,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:48,402 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:48,403 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:54:48,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:48,403 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:48,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:54:48,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:48,403 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:54:48,404 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:54:48,404 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:54:48,404 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:54:48,404 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:48,404 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:54:48,404 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:54:48,404 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:54:48,404 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. 2023-04-25 02:54:48,404 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. started... 2023-04-25 02:54:48,404 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. as already flushing 2023-04-25 02:54:48,404 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. 2023-04-25 02:54:48,404 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:54:48,404 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:54:48,404 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:48,404 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:54:48,404 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:54:48,410 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:48,410 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:48,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:48,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:48,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:48,411 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:48,411 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:54:48,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:48,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:48,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:48,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:48,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:48,412 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:54:48,412 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@254695c4[Count = 0] remaining members to acquire global barrier 2023-04-25 02:54:48,412 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:54:48,412 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:48,415 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:48,415 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:48,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:48,415 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:54:48,415 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:48,415 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:54:48,415 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:54:48,415 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:54:48,416 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:54:48,417 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:48,417 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:48,417 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:54:48,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:48,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:48,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:48,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:48,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:48,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:48,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:48,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:48,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:48,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:48,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:54:48,419 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:54:48,419 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:54:48,419 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:54:48,419 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:54:48,419 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:54:48,419 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:54:48,420 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:48,420 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:48,420 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:48,420 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:48,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:48,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:48,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:48,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:48,421 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:48,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:48,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:48,421 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:48,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:48,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:48,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:48,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:48,422 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:48,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:48,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:48,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:48,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:48,425 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:48,425 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:48,425 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:48,425 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:48,425 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:48,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:48,425 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:48,425 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:48,425 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:54:48,425 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:54:48,425 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:48,425 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:48,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:48,425 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:48,426 DEBUG [Thread-1512] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:54:48,426 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:48,426 DEBUG [Thread-1512] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:54:48,426 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:48,464 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:48,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59160 deadline: 1682391348461, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:48,468 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=663 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp/B/bebfc9606c914037beea7f4da2d95826 2023-04-25 02:54:48,478 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=663 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp/C/17c8516514d84ca69b3424137388e781 2023-04-25 02:54:48,481 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:48,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59214 deadline: 1682391348479, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:48,484 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp/A/9dfeafb5844843d2a4f4d1193640fc58 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A/9dfeafb5844843d2a4f4d1193640fc58 2023-04-25 02:54:48,492 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A/9dfeafb5844843d2a4f4d1193640fc58, entries=250, sequenceid=663, filesize=16.7 K 2023-04-25 02:54:48,493 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp/B/bebfc9606c914037beea7f4da2d95826 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B/bebfc9606c914037beea7f4da2d95826 2023-04-25 02:54:48,497 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B/bebfc9606c914037beea7f4da2d95826, entries=150, sequenceid=663, filesize=11.9 K 2023-04-25 02:54:48,498 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp/C/17c8516514d84ca69b3424137388e781 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C/17c8516514d84ca69b3424137388e781 2023-04-25 02:54:48,507 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C/17c8516514d84ca69b3424137388e781, entries=150, sequenceid=663, filesize=11.9 K 2023-04-25 02:54:48,508 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~154.31 KB/158010, heapSize ~437.34 KB/447840, currentSize=26.84 KB/27480 for 40cf532a35b33bde962f4f28e36e887b in 461ms, sequenceid=663, compaction requested=true 2023-04-25 02:54:48,508 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 40cf532a35b33bde962f4f28e36e887b: 2023-04-25 02:54:48,508 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 02:54:48,508 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-25 02:54:48,508 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-25 02:54:48,508 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 02:54:48,508 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 02:54:48,510 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 54068 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-25 02:54:48,510 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HStore(1912): 40cf532a35b33bde962f4f28e36e887b/A is initiating minor compaction (all files) 2023-04-25 02:54:48,510 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49188 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-25 02:54:48,510 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 40cf532a35b33bde962f4f28e36e887b/A in TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. 2023-04-25 02:54:48,510 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HStore(1912): 40cf532a35b33bde962f4f28e36e887b/B is initiating minor compaction (all files) 2023-04-25 02:54:48,510 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 40cf532a35b33bde962f4f28e36e887b/B in TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. 2023-04-25 02:54:48,510 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A/cbe8e6cba8e04939a8da94367ec0e635, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A/f80956a04fd44e29953e6f8db595c477, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A/818b8e53d66a4bc1b70ccda74e5a67a9, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A/9dfeafb5844843d2a4f4d1193640fc58] into tmpdir=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp, totalSize=52.8 K 2023-04-25 02:54:48,510 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B/47cbd8623b5b4bf6a7b97e67c55451fd, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B/6c05444a679b4b46a4b38a7569e2cf74, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B/e45a857578e9461382e39d5313efd5df, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B/bebfc9606c914037beea7f4da2d95826] into tmpdir=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp, totalSize=48.0 K 2023-04-25 02:54:48,511 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting cbe8e6cba8e04939a8da94367ec0e635, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=506, earliestPutTs=1682391286004 2023-04-25 02:54:48,511 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting 47cbd8623b5b4bf6a7b97e67c55451fd, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=506, earliestPutTs=1682391286004 2023-04-25 02:54:48,511 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting f80956a04fd44e29953e6f8db595c477, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=544, earliestPutTs=1682391286097 2023-04-25 02:54:48,511 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting 6c05444a679b4b46a4b38a7569e2cf74, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=544, earliestPutTs=1682391286097 2023-04-25 02:54:48,511 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting 818b8e53d66a4bc1b70ccda74e5a67a9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=556, earliestPutTs=1682391286714 2023-04-25 02:54:48,511 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting e45a857578e9461382e39d5313efd5df, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=556, earliestPutTs=1682391286714 2023-04-25 02:54:48,512 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting 9dfeafb5844843d2a4f4d1193640fc58, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=663, earliestPutTs=1682391287426 2023-04-25 02:54:48,512 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting bebfc9606c914037beea7f4da2d95826, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=663, earliestPutTs=1682391287426 2023-04-25 02:54:48,523 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] throttle.PressureAwareThroughputController(145): 40cf532a35b33bde962f4f28e36e887b#A#compaction#234 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 53.85 MB/second 2023-04-25 02:54:48,523 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] throttle.PressureAwareThroughputController(145): 40cf532a35b33bde962f4f28e36e887b#B#compaction#235 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-25 02:54:48,526 DEBUG [Thread-1512] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:54:48,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:54:48,530 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:54:48,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(9158): Flush requested on 40cf532a35b33bde962f4f28e36e887b 2023-04-25 02:54:48,542 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 40cf532a35b33bde962f4f28e36e887b 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-25 02:54:48,543 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 40cf532a35b33bde962f4f28e36e887b, store=A 2023-04-25 02:54:48,543 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:54:48,543 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 40cf532a35b33bde962f4f28e36e887b, store=B 2023-04-25 02:54:48,543 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:54:48,543 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 40cf532a35b33bde962f4f28e36e887b, store=C 2023-04-25 02:54:48,543 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:54:48,568 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:54:48,568 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:54:48,568 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:54:48,568 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:48,568 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:54:48,568 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:54:48,569 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:48,569 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:48,570 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:48,570 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:48,570 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:48,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:48,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:48,571 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:48,571 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:54:48,571 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:48,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:54:48,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:48,572 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:54:48,573 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp/A/15decf60ca8c4eebb169da7532e5b1b8 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A/15decf60ca8c4eebb169da7532e5b1b8 2023-04-25 02:54:48,578 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 40cf532a35b33bde962f4f28e36e887b/A of 40cf532a35b33bde962f4f28e36e887b into 15decf60ca8c4eebb169da7532e5b1b8(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 02:54:48,578 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 40cf532a35b33bde962f4f28e36e887b: 2023-04-25 02:54:48,578 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b., storeName=40cf532a35b33bde962f4f28e36e887b/A, priority=12, startTime=1682391288508; duration=0sec 2023-04-25 02:54:48,578 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 02:54:48,578 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-25 02:54:48,579 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49188 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-25 02:54:48,580 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HStore(1912): 40cf532a35b33bde962f4f28e36e887b/C is initiating minor compaction (all files) 2023-04-25 02:54:48,580 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 40cf532a35b33bde962f4f28e36e887b/C in TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. 2023-04-25 02:54:48,580 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:54:48,580 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:54:48,580 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C/7ee5c141bc454854a59b483fc5a6028d, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C/aa98107974264ee8b9ef7be771d8e5e5, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C/c8591ba6d2734211b1962f3142544a13, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C/17c8516514d84ca69b3424137388e781] into tmpdir=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp, totalSize=48.0 K 2023-04-25 02:54:48,580 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:54:48,580 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:48,580 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:54:48,580 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting 7ee5c141bc454854a59b483fc5a6028d, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=506, earliestPutTs=1682391286004 2023-04-25 02:54:48,581 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:54:48,581 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting aa98107974264ee8b9ef7be771d8e5e5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=544, earliestPutTs=1682391286097 2023-04-25 02:54:48,581 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:54:48,581 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. 2023-04-25 02:54:48,581 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. started... 2023-04-25 02:54:48,581 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. as already flushing 2023-04-25 02:54:48,581 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. 2023-04-25 02:54:48,581 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:54:48,581 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting c8591ba6d2734211b1962f3142544a13, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=556, earliestPutTs=1682391286714 2023-04-25 02:54:48,581 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:54:48,581 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:48,581 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:54:48,581 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:54:48,582 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting 17c8516514d84ca69b3424137388e781, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=663, earliestPutTs=1682391287426 2023-04-25 02:54:48,583 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=673 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp/A/bfad6c23c6114861a1006cf7314ee700 2023-04-25 02:54:48,586 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:48,586 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:48,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:48,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:48,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:48,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:48,586 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:48,587 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:54:48,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:48,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:48,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:48,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:48,588 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:54:48,588 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6ef85f89[Count = 0] remaining members to acquire global barrier 2023-04-25 02:54:48,588 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:54:48,588 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:48,591 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:48,591 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:48,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:48,591 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:54:48,591 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:54:48,591 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:54:48,594 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] throttle.PressureAwareThroughputController(145): 40cf532a35b33bde962f4f28e36e887b#C#compaction#238 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-25 02:54:48,594 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:48,594 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:54:48,595 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:48,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:48,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:48,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:48,596 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:54:48,596 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:48,596 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:54:48,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:48,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:48,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:48,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:48,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:48,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:48,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:48,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:54:48,599 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:54:48,599 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:54:48,599 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:54:48,599 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:54:48,599 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:54:48,599 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:54:48,600 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:48,600 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:48,600 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:48,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:48,601 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:48,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:48,601 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:48,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:48,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:48,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:48,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:48,601 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:48,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:48,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:48,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:48,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:48,603 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:48,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:48,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:48,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:48,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:48,605 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:48,605 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:48,605 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:48,605 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:54:48,605 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:48,605 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:48,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:48,605 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:48,605 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:54:48,605 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:54:48,605 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:48,605 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:48,606 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:48,606 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:48,606 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:48,606 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:48,606 DEBUG [Thread-1512] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:54:48,606 DEBUG [Thread-1512] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:54:48,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:48,615 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682391288598, max timestamp=1682391288613Num uniques 859; 2023-04-25 02:54:48,615 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682391288598, max timestamp=1682391288613Num uniques 859; 2023-04-25 02:54:48,615 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-25 02:54:48,615 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682391288598, max timestamp=1682391288613Num uniques 859; 2023-04-25 02:54:48,615 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-25 02:54:48,615 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-25 02:54:48,680 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1682391288660, max timestamp=1682391288678Num uniques 1018; 2023-04-25 02:54:48,680 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-25 02:54:48,681 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1682391288660, max timestamp=1682391288678Num uniques 1018; 2023-04-25 02:54:48,681 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1682391288660, max timestamp=1682391288678Num uniques 1018; 2023-04-25 02:54:48,681 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-25 02:54:48,681 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-25 02:54:48,698 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:48,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59216 deadline: 1682391348695, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:48,699 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:48,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 290 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59158 deadline: 1682391348697, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:48,701 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:48,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59186 deadline: 1682391348699, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:48,706 DEBUG [Thread-1512] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:54:48,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:54:48,708 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:54:48,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:54:48,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:54:48,709 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:54:48,709 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:48,709 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:54:48,709 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:54:48,710 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:48,710 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:48,712 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:48,712 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:48,712 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:48,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:48,712 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:48,712 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:54:48,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:48,713 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:48,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:54:48,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:48,713 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:54:48,713 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:54:48,713 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:54:48,713 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:54:48,713 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:48,713 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:54:48,714 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:54:48,714 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:54:48,714 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. 2023-04-25 02:54:48,714 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. started... 2023-04-25 02:54:48,714 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. as already flushing 2023-04-25 02:54:48,714 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. 2023-04-25 02:54:48,714 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:54:48,714 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:54:48,714 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:48,714 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:54:48,714 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:54:48,715 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:48,715 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:48,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:48,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:48,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:48,716 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:48,716 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:54:48,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:48,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:48,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:48,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:48,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:48,717 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:54:48,717 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@33ae24fb[Count = 0] remaining members to acquire global barrier 2023-04-25 02:54:48,717 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:54:48,717 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:48,718 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:48,718 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:48,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:48,719 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:54:48,719 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:54:48,719 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:48,719 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:54:48,719 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:54:48,720 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:48,720 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:54:48,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:48,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:48,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:48,720 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:48,720 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:54:48,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:48,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:48,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:48,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:48,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:48,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:48,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:48,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:54:48,722 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:54:48,722 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:54:48,723 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:54:48,723 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:54:48,723 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:54:48,723 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:54:48,725 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:48,725 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:48,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:48,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:48,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:48,725 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:48,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:48,725 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:48,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:48,726 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:48,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:48,726 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:48,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:48,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:48,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:48,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:48,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:48,727 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:48,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:48,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:48,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:48,729 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:48,729 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:48,729 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:48,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:48,729 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:48,730 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:54:48,730 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:54:48,730 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:48,730 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:48,730 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:48,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:48,730 DEBUG [Thread-1512] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:54:48,730 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:48,730 DEBUG [Thread-1512] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:54:48,730 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:48,730 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:48,730 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:48,730 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:48,801 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:48,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59216 deadline: 1682391348799, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:48,802 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:48,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 292 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59158 deadline: 1682391348800, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:48,804 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:48,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59186 deadline: 1682391348802, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:48,830 DEBUG [Thread-1512] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:54:48,831 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:54:48,832 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:54:48,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:54:48,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:54:48,833 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:54:48,833 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:48,834 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:54:48,834 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:54:48,834 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:48,834 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:48,835 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:48,835 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:48,835 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:48,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:48,835 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:48,835 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:54:48,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:48,836 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:48,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:54:48,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:48,836 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:54:48,836 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:54:48,836 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:54:48,836 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:54:48,837 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:48,837 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:54:48,837 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:54:48,837 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:54:48,837 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. 2023-04-25 02:54:48,837 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. started... 2023-04-25 02:54:48,837 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. as already flushing 2023-04-25 02:54:48,837 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. 2023-04-25 02:54:48,837 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:54:48,837 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:54:48,837 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:48,837 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:54:48,837 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:54:48,839 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:48,839 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:48,839 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:54:48,839 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:48,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:48,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:48,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:48,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:48,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:48,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:48,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:48,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:48,841 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:54:48,841 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@768fc2a8[Count = 0] remaining members to acquire global barrier 2023-04-25 02:54:48,841 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:54:48,841 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:48,843 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:48,843 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:48,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:48,843 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:54:48,843 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:48,843 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:54:48,843 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:54:48,843 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:54:48,845 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:48,845 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:54:48,845 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:48,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:48,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:48,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:48,845 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:54:48,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:48,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:48,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:48,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:48,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:48,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:48,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:48,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:54:48,848 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:54:48,848 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:54:48,848 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:54:48,848 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:54:48,848 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:54:48,848 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:54:48,849 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:48,849 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:48,849 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:48,850 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:48,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:48,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:48,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:48,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:48,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:48,850 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:48,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:48,850 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:48,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:48,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:48,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:48,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:48,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:48,851 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:48,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:48,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:48,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:48,853 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:48,853 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:48,853 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:48,853 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:54:48,853 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:54:48,853 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:48,853 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:54:48,853 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:48,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:48,854 DEBUG [Thread-1512] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:54:48,854 DEBUG [Thread-1512] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:54:48,853 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:48,853 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:48,854 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:48,854 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:48,854 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:48,854 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:48,854 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:48,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:48,954 DEBUG [Thread-1512] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:54:48,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:54:48,956 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:54:48,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:54:48,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:54:48,957 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:54:48,957 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:48,957 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:54:48,957 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:54:48,957 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:48,958 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:48,960 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:48,960 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:48,960 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:48,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:48,960 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:48,960 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:54:48,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:48,961 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:48,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:54:48,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:48,961 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:54:48,961 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:54:48,961 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:54:48,961 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:48,961 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:54:48,961 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:54:48,962 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:54:48,962 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. 2023-04-25 02:54:48,962 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. started... 2023-04-25 02:54:48,962 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. as already flushing 2023-04-25 02:54:48,962 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. 2023-04-25 02:54:48,962 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:54:48,962 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:54:48,962 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:48,962 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:54:48,962 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:54:48,963 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp/B/dba1c17f78134c6fb3ad8ccf325ee940 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B/dba1c17f78134c6fb3ad8ccf325ee940 2023-04-25 02:54:48,963 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:48,963 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:48,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:48,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:48,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:48,964 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:48,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:48,964 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:54:48,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:48,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:48,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:48,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:48,965 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:54:48,965 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@67768108[Count = 0] remaining members to acquire global barrier 2023-04-25 02:54:48,965 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:54:48,966 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:48,966 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:48,967 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:48,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:48,967 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:54:48,967 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:54:48,967 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:54:48,967 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:48,967 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:54:48,971 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:54:48,971 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:48,971 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:48,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:48,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:48,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:48,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:48,971 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:54:48,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:48,972 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 40cf532a35b33bde962f4f28e36e887b/B of 40cf532a35b33bde962f4f28e36e887b into dba1c17f78134c6fb3ad8ccf325ee940(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 02:54:48,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:48,972 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 40cf532a35b33bde962f4f28e36e887b: 2023-04-25 02:54:48,972 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b., storeName=40cf532a35b33bde962f4f28e36e887b/B, priority=12, startTime=1682391288508; duration=0sec 2023-04-25 02:54:48,972 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 02:54:48,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:48,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:48,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:48,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:48,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:54:48,974 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:54:48,974 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:54:48,974 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:54:48,974 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:54:48,974 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:54:48,974 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:54:48,975 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:48,975 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:48,975 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:48,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:48,975 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:48,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:48,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:48,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:48,976 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:48,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:48,976 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:48,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:48,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:48,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:48,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:48,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:48,978 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:48,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:48,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:48,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:48,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:48,980 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:48,980 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:48,980 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:48,980 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:48,980 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:48,981 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:48,981 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:48,980 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:48,981 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:48,981 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:54:48,981 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:54:48,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:48,981 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:48,981 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:48,981 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:54:48,981 DEBUG [Thread-1512] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:54:48,981 DEBUG [Thread-1512] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:54:48,981 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:48,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:49,004 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:49,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59216 deadline: 1682391349003, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:49,005 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:49,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 294 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59158 deadline: 1682391349004, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:49,006 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:49,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59186 deadline: 1682391349005, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:49,009 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=673 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp/B/08dca444144045e6be9e4676c13fdf9e 2023-04-25 02:54:49,034 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp/C/acc14600f8304afd87bda8566ab499eb as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C/acc14600f8304afd87bda8566ab499eb 2023-04-25 02:54:49,038 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 40cf532a35b33bde962f4f28e36e887b/C of 40cf532a35b33bde962f4f28e36e887b into acc14600f8304afd87bda8566ab499eb(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 02:54:49,038 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 40cf532a35b33bde962f4f28e36e887b: 2023-04-25 02:54:49,038 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b., storeName=40cf532a35b33bde962f4f28e36e887b/C, priority=12, startTime=1682391288508; duration=0sec 2023-04-25 02:54:49,038 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 02:54:49,081 DEBUG [Thread-1512] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:54:49,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:54:49,083 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:54:49,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:54:49,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:54:49,085 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:54:49,085 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:49,085 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:54:49,085 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:54:49,085 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:49,085 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:49,086 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:49,086 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:49,086 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:49,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:49,087 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:49,087 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:54:49,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:49,087 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:49,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:54:49,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:49,087 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:54:49,088 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:54:49,088 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:54:49,088 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:54:49,088 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:49,088 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:54:49,088 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:54:49,088 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:54:49,088 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. 2023-04-25 02:54:49,088 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. started... 2023-04-25 02:54:49,088 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. as already flushing 2023-04-25 02:54:49,088 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. 2023-04-25 02:54:49,088 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:54:49,088 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:54:49,089 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:49,089 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:54:49,089 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:54:49,092 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:49,092 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:49,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:49,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:49,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:49,092 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:49,093 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:54:49,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:49,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:49,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:49,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:49,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:49,094 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:54:49,094 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4b5e8e78[Count = 0] remaining members to acquire global barrier 2023-04-25 02:54:49,094 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:54:49,094 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:49,095 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:49,095 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:49,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:49,095 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:54:49,095 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:54:49,095 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:54:49,095 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:49,095 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:54:49,096 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:49,096 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:54:49,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:49,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:49,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:49,096 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:49,097 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:54:49,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:49,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:49,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:49,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:49,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:49,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:49,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:49,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:54:49,099 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:54:49,099 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:54:49,099 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:54:49,099 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:54:49,099 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:54:49,099 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:54:49,100 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:49,100 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:49,101 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:49,101 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:49,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:49,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:49,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:49,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:49,101 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:49,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:49,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:49,102 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:49,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:49,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:49,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:49,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:49,103 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:49,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:49,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:49,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:49,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:49,105 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:49,105 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:49,105 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:49,105 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:49,105 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:49,105 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:49,105 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:54:49,106 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:54:49,105 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:49,105 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:49,106 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:49,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:49,106 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:49,106 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:54:49,106 DEBUG [Thread-1512] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:54:49,106 DEBUG [Thread-1512] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:54:49,106 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:49,106 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:49,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:49,206 DEBUG [Thread-1512] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:54:49,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:54:49,208 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:54:49,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:54:49,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:54:49,209 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:54:49,209 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:49,209 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:54:49,209 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:54:49,210 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:49,210 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:49,211 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:49,211 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:49,211 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:49,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:49,211 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:49,211 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:54:49,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:49,212 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:49,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:54:49,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:49,212 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:54:49,212 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:54:49,212 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:54:49,212 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:54:49,212 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:49,213 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:54:49,213 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:54:49,213 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:54:49,213 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. 2023-04-25 02:54:49,213 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. started... 2023-04-25 02:54:49,213 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. as already flushing 2023-04-25 02:54:49,213 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. 2023-04-25 02:54:49,213 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:54:49,213 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:54:49,213 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:49,213 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:54:49,213 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:54:49,214 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:49,214 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:49,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:49,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:49,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:49,215 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:49,215 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:54:49,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:49,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:49,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:49,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:49,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:49,216 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:54:49,216 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@995e214[Count = 0] remaining members to acquire global barrier 2023-04-25 02:54:49,216 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:54:49,216 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:49,217 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:49,217 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:49,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:49,217 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:54:49,217 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:54:49,217 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:54:49,217 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:49,217 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:54:49,219 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:54:49,219 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:49,219 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:49,219 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:54:49,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:49,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:49,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:49,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:49,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:49,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:49,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:49,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:49,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:49,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:49,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:54:49,222 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:54:49,222 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:54:49,222 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:54:49,222 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:54:49,222 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:54:49,222 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:54:49,223 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:49,223 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:49,223 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:49,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:49,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:49,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:49,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:49,223 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:49,224 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:49,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:49,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:49,224 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:49,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:49,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:49,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:49,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:49,225 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:49,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:49,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:49,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:49,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:49,227 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:49,227 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:49,227 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:49,227 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:49,227 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:54:49,227 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:54:49,227 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:49,227 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:49,227 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:49,227 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:49,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:49,227 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:49,227 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:49,227 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:49,227 DEBUG [Thread-1512] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:54:49,227 DEBUG [Thread-1512] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:54:49,227 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:49,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:49,307 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:49,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59216 deadline: 1682391349306, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:49,307 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:49,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 296 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59158 deadline: 1682391349306, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:49,310 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:49,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59186 deadline: 1682391349308, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:49,327 DEBUG [Thread-1512] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:54:49,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:54:49,329 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:54:49,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:54:49,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:54:49,330 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:54:49,330 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:49,331 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:54:49,331 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:54:49,331 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:49,331 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:49,332 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:49,332 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:49,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:49,332 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:49,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:49,332 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:49,332 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:54:49,333 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:49,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:54:49,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:49,333 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:54:49,333 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:54:49,333 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:54:49,333 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:54:49,333 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:49,333 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:54:49,334 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:54:49,334 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:54:49,334 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. 2023-04-25 02:54:49,334 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. started... 2023-04-25 02:54:49,334 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. as already flushing 2023-04-25 02:54:49,334 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. 2023-04-25 02:54:49,334 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:54:49,334 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:54:49,334 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:49,334 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:54:49,334 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:54:49,336 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:49,336 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:49,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:49,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:49,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:49,336 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:49,336 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:54:49,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:49,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:49,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:49,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:49,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:49,338 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:54:49,338 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@187eb0c[Count = 0] remaining members to acquire global barrier 2023-04-25 02:54:49,338 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:54:49,338 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:49,339 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:49,339 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:49,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:49,339 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:54:49,339 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:54:49,339 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:49,339 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:54:49,339 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:54:49,340 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:49,340 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:54:49,340 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:49,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:49,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:49,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:49,340 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:54:49,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:49,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:49,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:49,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:49,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:49,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:49,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:49,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:54:49,342 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:54:49,342 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:54:49,342 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:54:49,342 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:54:49,343 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:54:49,343 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:54:49,350 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:49,350 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:49,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:49,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:49,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:49,350 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:49,350 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:49,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:49,350 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:49,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:49,351 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:49,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:49,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:49,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:49,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:49,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:49,352 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:49,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:49,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:49,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:49,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:49,354 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:49,354 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:49,354 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:49,354 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:49,354 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:49,354 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:49,354 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:54:49,354 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:49,354 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:49,354 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:54:49,354 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:54:49,354 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:49,354 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:49,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:49,354 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:49,354 DEBUG [Thread-1512] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:54:49,354 DEBUG [Thread-1512] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:54:49,354 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:49,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:49,421 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=673 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp/C/cd8f31c2e6a94246a6a3dc333e52e680 2023-04-25 02:54:49,426 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp/A/bfad6c23c6114861a1006cf7314ee700 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A/bfad6c23c6114861a1006cf7314ee700 2023-04-25 02:54:49,432 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A/bfad6c23c6114861a1006cf7314ee700, entries=200, sequenceid=673, filesize=14.3 K 2023-04-25 02:54:49,433 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp/B/08dca444144045e6be9e4676c13fdf9e as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B/08dca444144045e6be9e4676c13fdf9e 2023-04-25 02:54:49,437 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B/08dca444144045e6be9e4676c13fdf9e, entries=150, sequenceid=673, filesize=11.9 K 2023-04-25 02:54:49,440 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp/C/cd8f31c2e6a94246a6a3dc333e52e680 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C/cd8f31c2e6a94246a6a3dc333e52e680 2023-04-25 02:54:49,447 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C/cd8f31c2e6a94246a6a3dc333e52e680, entries=150, sequenceid=673, filesize=11.9 K 2023-04-25 02:54:49,448 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 40cf532a35b33bde962f4f28e36e887b in 906ms, sequenceid=673, compaction requested=false 2023-04-25 02:54:49,448 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 40cf532a35b33bde962f4f28e36e887b: 2023-04-25 02:54:49,454 DEBUG [Thread-1512] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:54:49,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:54:49,456 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:54:49,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:54:49,459 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:54:49,459 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:54:49,459 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:49,459 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:54:49,459 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:54:49,459 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:49,459 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:49,460 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:49,460 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:49,460 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:49,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:49,461 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:49,461 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:54:49,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:49,461 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:49,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:54:49,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:49,461 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:54:49,462 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:54:49,462 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:54:49,462 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:54:49,462 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:49,462 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:54:49,467 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:54:49,467 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:54:49,467 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. 2023-04-25 02:54:49,467 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. started... 2023-04-25 02:54:49,467 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 40cf532a35b33bde962f4f28e36e887b 3/3 column families, dataSize=134.18 KB heapSize=379.45 KB 2023-04-25 02:54:49,467 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 40cf532a35b33bde962f4f28e36e887b, store=A 2023-04-25 02:54:49,467 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 02:54:49,468 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 40cf532a35b33bde962f4f28e36e887b, store=B 2023-04-25 02:54:49,468 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 02:54:49,468 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 40cf532a35b33bde962f4f28e36e887b, store=C 2023-04-25 02:54:49,468 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 02:54:49,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(9158): Flush requested on 40cf532a35b33bde962f4f28e36e887b 2023-04-25 02:54:49,474 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. as already flushing 2023-04-25 02:54:49,511 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:49,511 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:49,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59160 deadline: 1682391349507, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:49,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59214 deadline: 1682391349507, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:49,614 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:49,614 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:49,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59160 deadline: 1682391349613, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:49,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59214 deadline: 1682391349613, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:49,810 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:49,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 298 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59158 deadline: 1682391349808, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:49,810 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:49,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59216 deadline: 1682391349809, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:49,813 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:49,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 201 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59186 deadline: 1682391349811, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:49,818 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:49,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59214 deadline: 1682391349816, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:49,818 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:49,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59160 deadline: 1682391349816, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:49,890 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=729 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp/A/e5225d64e6aa4b3f96fa5d19454ef470 2023-04-25 02:54:50,122 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:50,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59214 deadline: 1682391350120, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:50,122 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:50,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 216 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59160 deadline: 1682391350120, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:50,301 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=729 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp/B/74e8ea961ae2425c8c7f9403bf29ad90 2023-04-25 02:54:50,313 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=729 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp/C/5df0c28533b544ecaac582e943da5814 2023-04-25 02:54:50,318 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp/A/e5225d64e6aa4b3f96fa5d19454ef470 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A/e5225d64e6aa4b3f96fa5d19454ef470 2023-04-25 02:54:50,321 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A/e5225d64e6aa4b3f96fa5d19454ef470, entries=150, sequenceid=729, filesize=11.9 K 2023-04-25 02:54:50,322 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp/B/74e8ea961ae2425c8c7f9403bf29ad90 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B/74e8ea961ae2425c8c7f9403bf29ad90 2023-04-25 02:54:50,326 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B/74e8ea961ae2425c8c7f9403bf29ad90, entries=150, sequenceid=729, filesize=11.9 K 2023-04-25 02:54:50,326 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp/C/5df0c28533b544ecaac582e943da5814 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C/5df0c28533b544ecaac582e943da5814 2023-04-25 02:54:50,330 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C/5df0c28533b544ecaac582e943da5814, entries=150, sequenceid=729, filesize=11.9 K 2023-04-25 02:54:50,330 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~379.41 KB/388512, currentSize=53.67 KB/54960 for 40cf532a35b33bde962f4f28e36e887b in 863ms, sequenceid=729, compaction requested=true 2023-04-25 02:54:50,330 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 40cf532a35b33bde962f4f28e36e887b: 2023-04-25 02:54:50,331 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. 2023-04-25 02:54:50,331 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:54:50,331 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:54:50,331 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:50,331 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:54:50,331 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:54:50,332 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:50,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:50,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:50,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:50,332 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:50,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:50,333 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:50,333 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:54:50,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:50,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:50,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:50,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:50,334 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:54:50,334 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@52aa54f5[Count = 0] remaining members to acquire global barrier 2023-04-25 02:54:50,334 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:54:50,334 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:50,336 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:50,336 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:50,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:50,336 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:54:50,336 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:54:50,336 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:54:50,336 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:50,336 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:54:50,337 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:50,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:50,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:50,337 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:54:50,337 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:50,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:50,337 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:54:50,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:50,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:50,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:50,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:50,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:50,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:50,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:50,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:54:50,341 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:54:50,341 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:54:50,341 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:54:50,341 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:54:50,341 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:54:50,341 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:54:50,342 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:50,342 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:50,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:50,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:50,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:50,342 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:50,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:50,342 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:50,343 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:50,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:50,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:50,343 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:50,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:50,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:50,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:50,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:50,344 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:50,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:50,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:50,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:50,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:50,347 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:50,347 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:50,347 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:50,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:50,347 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:50,347 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:50,347 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:50,347 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:50,347 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:54:50,347 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:54:50,347 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:50,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:50,347 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:50,347 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:50,347 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:50,347 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:50,347 DEBUG [Thread-1512] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:54:50,348 DEBUG [Thread-1512] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:54:50,448 DEBUG [Thread-1512] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:54:50,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:54:50,449 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:54:50,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:54:50,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:54:50,451 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:54:50,451 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:50,451 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:54:50,451 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:54:50,451 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:50,451 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:50,454 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:50,454 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:50,454 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:50,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:50,454 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:50,454 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:54:50,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:50,454 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:50,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:54:50,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:50,455 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:54:50,455 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:54:50,455 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:54:50,455 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:54:50,455 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:50,455 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:54:50,455 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:54:50,455 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:54:50,455 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. 2023-04-25 02:54:50,456 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. started... 2023-04-25 02:54:50,456 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 40cf532a35b33bde962f4f28e36e887b 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-25 02:54:50,456 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 40cf532a35b33bde962f4f28e36e887b, store=A 2023-04-25 02:54:50,456 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:54:50,456 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 40cf532a35b33bde962f4f28e36e887b, store=B 2023-04-25 02:54:50,456 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:54:50,456 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 40cf532a35b33bde962f4f28e36e887b, store=C 2023-04-25 02:54:50,456 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:54:50,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(9158): Flush requested on 40cf532a35b33bde962f4f28e36e887b 2023-04-25 02:54:50,626 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. as already flushing 2023-04-25 02:54:50,654 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:50,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59214 deadline: 1682391350652, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:50,655 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682391290645, max timestamp=1682391290653Num uniques 859; 2023-04-25 02:54:50,655 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682391290645, max timestamp=1682391290653Num uniques 859; 2023-04-25 02:54:50,655 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-25 02:54:50,655 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682391290645, max timestamp=1682391290653Num uniques 859; 2023-04-25 02:54:50,656 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-25 02:54:50,656 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-25 02:54:50,701 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:50,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 243 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59160 deadline: 1682391350699, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:50,757 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:50,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59214 deadline: 1682391350755, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:50,804 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:50,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 245 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59160 deadline: 1682391350802, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:50,816 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:50,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 300 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59158 deadline: 1682391350814, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:50,817 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:50,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59216 deadline: 1682391350816, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:50,825 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:50,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59186 deadline: 1682391350823, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:50,866 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=740 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp/A/2399b71f9c3344ad9483c17ec0757716 2023-04-25 02:54:50,960 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:50,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59214 deadline: 1682391350959, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:51,009 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:51,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 247 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59160 deadline: 1682391351005, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:51,265 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:51,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59214 deadline: 1682391351263, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:51,275 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=740 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp/B/3c879b9204e144a1afb40fee4449ae89 2023-04-25 02:54:51,312 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:51,313 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 249 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59160 deadline: 1682391351311, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:51,685 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=740 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp/C/0dee40da444048a098e3730c0e1f7e28 2023-04-25 02:54:51,690 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp/A/2399b71f9c3344ad9483c17ec0757716 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A/2399b71f9c3344ad9483c17ec0757716 2023-04-25 02:54:51,694 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A/2399b71f9c3344ad9483c17ec0757716, entries=150, sequenceid=740, filesize=11.9 K 2023-04-25 02:54:51,695 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp/B/3c879b9204e144a1afb40fee4449ae89 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B/3c879b9204e144a1afb40fee4449ae89 2023-04-25 02:54:51,699 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B/3c879b9204e144a1afb40fee4449ae89, entries=150, sequenceid=740, filesize=11.9 K 2023-04-25 02:54:51,700 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp/C/0dee40da444048a098e3730c0e1f7e28 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C/0dee40da444048a098e3730c0e1f7e28 2023-04-25 02:54:51,704 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C/0dee40da444048a098e3730c0e1f7e28, entries=150, sequenceid=740, filesize=11.9 K 2023-04-25 02:54:51,708 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 40cf532a35b33bde962f4f28e36e887b in 1252ms, sequenceid=740, compaction requested=true 2023-04-25 02:54:51,708 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 40cf532a35b33bde962f4f28e36e887b: 2023-04-25 02:54:51,709 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. 2023-04-25 02:54:51,709 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:54:51,709 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:54:51,709 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:51,709 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:54:51,709 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:54:51,710 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:51,710 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:51,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:51,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:51,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:51,711 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:51,711 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:54:51,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:51,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:51,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:51,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:51,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:51,712 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:54:51,712 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6db83d96[Count = 0] remaining members to acquire global barrier 2023-04-25 02:54:51,712 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:54:51,712 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:51,713 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:51,713 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:51,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:51,713 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:54:51,713 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:54:51,713 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:54:51,714 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:51,714 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:54:51,715 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:54:51,715 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:51,715 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:51,715 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:54:51,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:51,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:51,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:51,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:51,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:51,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:51,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:51,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:51,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:51,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:51,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:54:51,717 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:54:51,717 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:54:51,717 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:54:51,717 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:54:51,717 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:54:51,717 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:54:51,720 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:51,720 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:51,720 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:51,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:51,720 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:51,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:51,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:51,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:51,720 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:51,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:51,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:51,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:51,721 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:51,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:51,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:51,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:51,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:51,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:51,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:51,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:51,725 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:51,727 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:51,727 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:51,727 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:51,727 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:51,727 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:51,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:51,727 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:51,727 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:54:51,727 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:54:51,727 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:51,727 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:51,727 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:51,727 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:51,727 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:51,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:51,727 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:51,727 DEBUG [Thread-1512] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:54:51,727 DEBUG [Thread-1512] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:54:51,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(9158): Flush requested on 40cf532a35b33bde962f4f28e36e887b 2023-04-25 02:54:51,773 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 40cf532a35b33bde962f4f28e36e887b 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-04-25 02:54:51,773 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 40cf532a35b33bde962f4f28e36e887b, store=A 2023-04-25 02:54:51,773 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 02:54:51,773 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 40cf532a35b33bde962f4f28e36e887b, store=B 2023-04-25 02:54:51,773 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 02:54:51,773 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 40cf532a35b33bde962f4f28e36e887b, store=C 2023-04-25 02:54:51,773 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 02:54:51,781 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=777 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp/A/2b5a29dae9074175b6caedaebc256abc 2023-04-25 02:54:51,791 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=777 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp/B/ae0e1db7e0874f4cae02d06057d671dd 2023-04-25 02:54:51,798 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:51,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59214 deadline: 1682391351796, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:51,820 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:51,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 251 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59160 deadline: 1682391351817, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:51,827 DEBUG [Thread-1512] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:54:51,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:54:51,829 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:54:51,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:54:51,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:54:51,833 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:54:51,833 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:51,833 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:54:51,833 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:54:51,833 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:51,834 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:51,835 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:51,835 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:51,835 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:51,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:51,835 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:51,835 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:54:51,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:51,835 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:51,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:54:51,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:51,836 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:54:51,836 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:54:51,836 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:54:51,836 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:54:51,836 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:51,836 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:54:51,836 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:54:51,836 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:54:51,836 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. 2023-04-25 02:54:51,836 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. started... 2023-04-25 02:54:51,836 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. as already flushing 2023-04-25 02:54:51,836 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. 2023-04-25 02:54:51,836 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:54:51,837 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:54:51,837 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:51,837 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:54:51,837 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:54:51,838 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:51,838 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:51,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:51,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:51,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:51,838 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:51,838 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:54:51,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:51,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:51,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:51,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:51,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:51,840 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:54:51,840 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4b1bf777[Count = 0] remaining members to acquire global barrier 2023-04-25 02:54:51,840 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:54:51,840 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:51,842 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:51,842 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:51,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:51,842 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:54:51,842 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:54:51,842 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:54:51,842 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:51,842 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:54:51,844 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:51,844 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:54:51,844 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:51,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:51,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:51,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:51,844 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:54:51,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:51,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:51,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:51,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:51,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:51,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:51,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:51,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:54:51,846 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:54:51,846 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:54:51,846 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:54:51,846 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:54:51,846 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:54:51,846 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:54:51,848 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:51,848 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:51,848 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:51,848 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:51,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:51,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:51,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:51,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:51,848 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:51,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:51,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:51,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:51,848 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:51,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:51,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:51,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:51,849 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:51,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:51,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:51,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:51,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:51,855 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:51,855 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:51,855 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:51,855 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:51,855 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:51,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:51,855 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:54:51,855 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:54:51,855 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:54:51,855 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:51,856 DEBUG [Thread-1512] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:54:51,855 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:51,855 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:51,856 DEBUG [Thread-1512] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:54:51,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:51,856 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:51,856 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:51,856 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:51,856 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:51,901 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:51,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59214 deadline: 1682391351899, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:51,956 DEBUG [Thread-1512] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:54:51,956 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:54:51,958 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:54:51,959 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:54:51,959 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:54:51,959 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:54:51,959 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:51,959 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:54:51,959 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:54:51,959 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:51,959 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:51,961 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:51,961 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:51,961 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:51,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:51,961 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:51,961 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:54:51,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:51,961 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:51,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:54:51,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:51,962 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:54:51,962 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:54:51,962 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:54:51,962 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:54:51,962 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:51,962 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:54:51,962 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:54:51,962 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:54:51,962 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. 2023-04-25 02:54:51,962 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. started... 2023-04-25 02:54:51,963 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. as already flushing 2023-04-25 02:54:51,963 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. 2023-04-25 02:54:51,963 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:54:51,963 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:54:51,963 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:51,963 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:54:51,963 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:54:51,964 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:51,964 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:51,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:51,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:51,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:51,964 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:51,965 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:54:51,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:51,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:51,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:51,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:51,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:51,966 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:54:51,966 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@75faaa93[Count = 0] remaining members to acquire global barrier 2023-04-25 02:54:51,966 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:54:51,966 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:51,967 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:51,967 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:51,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:51,967 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:54:51,967 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:54:51,967 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:51,967 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:54:51,967 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:54:51,970 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:54:51,970 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:51,970 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:51,970 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:54:51,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:51,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:51,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:51,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:51,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:51,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:51,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:51,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:51,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:51,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:51,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:54:51,972 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:54:51,972 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:54:51,972 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:54:51,972 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:54:51,972 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:54:51,972 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:54:51,974 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:51,974 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:51,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:51,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:51,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:51,974 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:51,974 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:51,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:51,974 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:51,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:51,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:51,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:51,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:51,974 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:51,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:51,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:51,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:51,976 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:51,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:51,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:51,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:51,978 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:51,978 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:51,978 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:51,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:51,978 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:54:51,978 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:54:51,978 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:51,978 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:51,978 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:51,978 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:51,978 DEBUG [Thread-1512] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:54:51,978 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:51,978 DEBUG [Thread-1512] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:54:51,978 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:51,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:51,978 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:51,978 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:51,978 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:52,078 DEBUG [Thread-1512] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:54:52,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:54:52,080 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:54:52,081 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:54:52,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:54:52,082 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:54:52,082 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:52,082 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:54:52,082 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:54:52,082 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:52,082 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:52,083 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:52,083 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:52,083 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:52,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:52,084 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:52,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:52,084 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:54:52,084 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:52,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:54:52,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:52,085 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:54:52,085 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:54:52,085 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:54:52,085 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:54:52,085 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:52,085 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:54:52,085 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:54:52,085 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:54:52,085 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. 2023-04-25 02:54:52,085 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. started... 2023-04-25 02:54:52,085 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. as already flushing 2023-04-25 02:54:52,085 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. 2023-04-25 02:54:52,086 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:54:52,086 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:54:52,086 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:52,086 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:54:52,086 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:54:52,088 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:52,088 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:52,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:52,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:52,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:52,088 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:52,088 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:54:52,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:52,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:52,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:52,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:52,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:52,090 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:54:52,090 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@75f75525[Count = 0] remaining members to acquire global barrier 2023-04-25 02:54:52,090 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:54:52,090 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:52,091 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:52,091 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:52,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:52,091 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:54:52,091 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:54:52,091 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:54:52,093 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:52,093 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:54:52,094 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:54:52,094 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:52,094 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:52,094 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:54:52,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:52,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:52,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:52,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:52,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:52,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:52,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:52,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:52,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:52,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:52,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:54:52,097 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:54:52,097 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:54:52,097 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:54:52,097 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:54:52,097 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:54:52,097 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:54:52,098 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:52,098 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:52,098 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:52,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:52,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:52,098 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:52,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:52,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:52,099 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:52,099 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:52,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:52,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:52,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:52,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:52,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:52,100 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:52,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:52,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:52,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:52,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:52,103 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:52,103 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:52,103 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:52,103 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:52,103 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:52,103 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:52,103 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:52,103 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:52,103 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:54:52,103 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:54:52,103 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:54:52,103 DEBUG [Thread-1512] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:54:52,103 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:52,103 DEBUG [Thread-1512] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:54:52,103 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:52,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:52,103 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:52,104 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:52,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:52,105 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:52,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59214 deadline: 1682391352103, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:52,203 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=777 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp/C/5c76b8d71c534b07946f1548b5df5a4f 2023-04-25 02:54:52,203 DEBUG [Thread-1512] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:54:52,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:54:52,206 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:54:52,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:54:52,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:54:52,207 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:54:52,207 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:52,207 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:54:52,207 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:54:52,207 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:52,208 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:52,208 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp/A/2b5a29dae9074175b6caedaebc256abc as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A/2b5a29dae9074175b6caedaebc256abc 2023-04-25 02:54:52,209 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:52,209 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:52,209 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:52,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:52,209 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:52,209 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:54:52,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:52,209 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:52,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:54:52,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:52,210 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:54:52,210 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:54:52,210 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:54:52,210 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:52,210 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:54:52,210 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:54:52,211 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:54:52,211 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. 2023-04-25 02:54:52,211 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. started... 2023-04-25 02:54:52,211 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. as already flushing 2023-04-25 02:54:52,211 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. 2023-04-25 02:54:52,211 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:54:52,211 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:54:52,211 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:52,211 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:54:52,211 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:54:52,212 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A/2b5a29dae9074175b6caedaebc256abc, entries=200, sequenceid=777, filesize=14.3 K 2023-04-25 02:54:52,212 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:52,213 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:52,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:52,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:52,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:52,213 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp/B/ae0e1db7e0874f4cae02d06057d671dd as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B/ae0e1db7e0874f4cae02d06057d671dd 2023-04-25 02:54:52,213 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:52,213 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:54:52,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:52,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:52,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:52,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:52,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:52,215 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:54:52,215 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:54:52,215 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@590f8280[Count = 0] remaining members to acquire global barrier 2023-04-25 02:54:52,215 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:52,216 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:52,216 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:52,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:52,216 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:54:52,216 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:54:52,216 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:54:52,216 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:52,216 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:54:52,217 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B/ae0e1db7e0874f4cae02d06057d671dd, entries=150, sequenceid=777, filesize=11.9 K 2023-04-25 02:54:52,217 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp/C/5c76b8d71c534b07946f1548b5df5a4f as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C/5c76b8d71c534b07946f1548b5df5a4f 2023-04-25 02:54:52,219 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:54:52,219 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:52,219 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:52,219 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:54:52,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:52,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:52,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:52,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:52,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:52,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:52,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:52,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:52,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:52,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:52,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:54:52,222 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:54:52,222 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:54:52,222 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:54:52,222 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:54:52,222 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:54:52,222 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:54:52,223 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:52,223 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:52,223 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:52,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:52,223 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:52,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:52,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:52,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:52,224 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:52,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:52,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:52,224 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:52,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:52,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:52,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:52,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:52,225 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C/5c76b8d71c534b07946f1548b5df5a4f, entries=150, sequenceid=777, filesize=11.9 K 2023-04-25 02:54:52,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:52,226 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=46.96 KB/48090 for 40cf532a35b33bde962f4f28e36e887b in 453ms, sequenceid=777, compaction requested=true 2023-04-25 02:54:52,226 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:52,226 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 40cf532a35b33bde962f4f28e36e887b: 2023-04-25 02:54:52,226 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 02:54:52,226 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-25 02:54:52,226 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 02:54:52,226 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-25 02:54:52,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:52,227 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 02:54:52,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:52,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:52,229 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61510 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-25 02:54:52,229 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 66390 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-25 02:54:52,229 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HStore(1912): 40cf532a35b33bde962f4f28e36e887b/B is initiating minor compaction (all files) 2023-04-25 02:54:52,229 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HStore(1912): 40cf532a35b33bde962f4f28e36e887b/A is initiating minor compaction (all files) 2023-04-25 02:54:52,229 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 40cf532a35b33bde962f4f28e36e887b/B in TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. 2023-04-25 02:54:52,229 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 40cf532a35b33bde962f4f28e36e887b/A in TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. 2023-04-25 02:54:52,229 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B/dba1c17f78134c6fb3ad8ccf325ee940, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B/08dca444144045e6be9e4676c13fdf9e, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B/74e8ea961ae2425c8c7f9403bf29ad90, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B/3c879b9204e144a1afb40fee4449ae89, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B/ae0e1db7e0874f4cae02d06057d671dd] into tmpdir=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp, totalSize=60.1 K 2023-04-25 02:54:52,229 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A/15decf60ca8c4eebb169da7532e5b1b8, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A/bfad6c23c6114861a1006cf7314ee700, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A/e5225d64e6aa4b3f96fa5d19454ef470, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A/2399b71f9c3344ad9483c17ec0757716, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A/2b5a29dae9074175b6caedaebc256abc] into tmpdir=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp, totalSize=64.8 K 2023-04-25 02:54:52,229 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting dba1c17f78134c6fb3ad8ccf325ee940, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=663, earliestPutTs=1682391287426 2023-04-25 02:54:52,229 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting 15decf60ca8c4eebb169da7532e5b1b8, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=663, earliestPutTs=1682391287426 2023-04-25 02:54:52,230 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting 08dca444144045e6be9e4676c13fdf9e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=673, earliestPutTs=1682391288055 2023-04-25 02:54:52,230 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:52,230 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:52,230 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:52,230 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:52,230 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:54:52,230 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:54:52,230 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting bfad6c23c6114861a1006cf7314ee700, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=673, earliestPutTs=1682391288051 2023-04-25 02:54:52,230 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:52,230 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:52,230 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:52,230 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:52,230 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:52,230 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:52,230 DEBUG [Thread-1512] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:54:52,230 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting 74e8ea961ae2425c8c7f9403bf29ad90, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=729, earliestPutTs=1682391288692 2023-04-25 02:54:52,230 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:52,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:52,230 DEBUG [Thread-1512] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:54:52,230 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:54:52,230 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting e5225d64e6aa4b3f96fa5d19454ef470, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=729, earliestPutTs=1682391288692 2023-04-25 02:54:52,231 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:52,231 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting 3c879b9204e144a1afb40fee4449ae89, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=740, earliestPutTs=1682391289493 2023-04-25 02:54:52,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:52,231 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting 2399b71f9c3344ad9483c17ec0757716, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=740, earliestPutTs=1682391289493 2023-04-25 02:54:52,231 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting ae0e1db7e0874f4cae02d06057d671dd, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=777, earliestPutTs=1682391290695 2023-04-25 02:54:52,231 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting 2b5a29dae9074175b6caedaebc256abc, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=777, earliestPutTs=1682391290685 2023-04-25 02:54:52,239 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] throttle.PressureAwareThroughputController(145): 40cf532a35b33bde962f4f28e36e887b#A#compaction#249 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 53.85 MB/second 2023-04-25 02:54:52,241 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] throttle.PressureAwareThroughputController(145): 40cf532a35b33bde962f4f28e36e887b#B#compaction#250 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-25 02:54:52,250 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp/A/6d781aadadbf44b98ee0c11862c78b54 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A/6d781aadadbf44b98ee0c11862c78b54 2023-04-25 02:54:52,255 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 40cf532a35b33bde962f4f28e36e887b/A of 40cf532a35b33bde962f4f28e36e887b into 6d781aadadbf44b98ee0c11862c78b54(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 02:54:52,255 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 40cf532a35b33bde962f4f28e36e887b: 2023-04-25 02:54:52,255 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b., storeName=40cf532a35b33bde962f4f28e36e887b/A, priority=11, startTime=1682391292226; duration=0sec 2023-04-25 02:54:52,255 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 02:54:52,255 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-25 02:54:52,257 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61510 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-25 02:54:52,257 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HStore(1912): 40cf532a35b33bde962f4f28e36e887b/C is initiating minor compaction (all files) 2023-04-25 02:54:52,257 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 40cf532a35b33bde962f4f28e36e887b/C in TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. 2023-04-25 02:54:52,257 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C/acc14600f8304afd87bda8566ab499eb, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C/cd8f31c2e6a94246a6a3dc333e52e680, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C/5df0c28533b544ecaac582e943da5814, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C/0dee40da444048a098e3730c0e1f7e28, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C/5c76b8d71c534b07946f1548b5df5a4f] into tmpdir=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp, totalSize=60.1 K 2023-04-25 02:54:52,257 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting acc14600f8304afd87bda8566ab499eb, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=663, earliestPutTs=1682391287426 2023-04-25 02:54:52,258 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting cd8f31c2e6a94246a6a3dc333e52e680, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=673, earliestPutTs=1682391288055 2023-04-25 02:54:52,258 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting 5df0c28533b544ecaac582e943da5814, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=729, earliestPutTs=1682391288692 2023-04-25 02:54:52,258 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting 0dee40da444048a098e3730c0e1f7e28, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=740, earliestPutTs=1682391289493 2023-04-25 02:54:52,258 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting 5c76b8d71c534b07946f1548b5df5a4f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=777, earliestPutTs=1682391290695 2023-04-25 02:54:52,262 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp/B/76ce613289ae4db386e81e6e16d7c4ce as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B/76ce613289ae4db386e81e6e16d7c4ce 2023-04-25 02:54:52,267 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] throttle.PressureAwareThroughputController(145): 40cf532a35b33bde962f4f28e36e887b#C#compaction#251 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-25 02:54:52,269 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 40cf532a35b33bde962f4f28e36e887b/B of 40cf532a35b33bde962f4f28e36e887b into 76ce613289ae4db386e81e6e16d7c4ce(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 02:54:52,269 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 40cf532a35b33bde962f4f28e36e887b: 2023-04-25 02:54:52,269 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b., storeName=40cf532a35b33bde962f4f28e36e887b/B, priority=11, startTime=1682391292226; duration=0sec 2023-04-25 02:54:52,270 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 02:54:52,330 DEBUG [Thread-1512] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:54:52,331 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:54:52,333 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:54:52,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:54:52,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:54:52,334 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:54:52,334 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:52,334 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:54:52,334 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:54:52,334 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:52,334 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:52,339 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:52,339 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:52,339 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:52,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:52,340 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:52,340 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:54:52,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:52,340 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:52,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:54:52,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:52,341 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:54:52,341 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:54:52,341 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:54:52,341 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:54:52,341 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:52,341 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:54:52,341 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:54:52,341 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:54:52,341 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. 2023-04-25 02:54:52,341 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. started... 2023-04-25 02:54:52,341 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 40cf532a35b33bde962f4f28e36e887b 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-25 02:54:52,342 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 40cf532a35b33bde962f4f28e36e887b, store=A 2023-04-25 02:54:52,342 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:54:52,342 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 40cf532a35b33bde962f4f28e36e887b, store=B 2023-04-25 02:54:52,342 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:54:52,342 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 40cf532a35b33bde962f4f28e36e887b, store=C 2023-04-25 02:54:52,342 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:54:52,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(9158): Flush requested on 40cf532a35b33bde962f4f28e36e887b 2023-04-25 02:54:52,408 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. as already flushing 2023-04-25 02:54:52,457 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682391292445, max timestamp=1682391292456Num uniques 859; 2023-04-25 02:54:52,457 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682391292445, max timestamp=1682391292456Num uniques 859; 2023-04-25 02:54:52,457 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-25 02:54:52,457 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-25 02:54:52,458 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682391292445, max timestamp=1682391292456Num uniques 859; 2023-04-25 02:54:52,458 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-25 02:54:52,516 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:52,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 219 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59214 deadline: 1682391352514, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:52,617 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:52,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 221 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59214 deadline: 1682391352617, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:52,677 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp/C/7c2a1f49b23a4fe08ab8c2567609d5aa as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C/7c2a1f49b23a4fe08ab8c2567609d5aa 2023-04-25 02:54:52,681 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 40cf532a35b33bde962f4f28e36e887b/C of 40cf532a35b33bde962f4f28e36e887b into 7c2a1f49b23a4fe08ab8c2567609d5aa(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 02:54:52,681 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 40cf532a35b33bde962f4f28e36e887b: 2023-04-25 02:54:52,681 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b., storeName=40cf532a35b33bde962f4f28e36e887b/C, priority=11, startTime=1682391292227; duration=0sec 2023-04-25 02:54:52,681 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 02:54:52,750 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=789 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp/A/c46e0fe7a3be483e838d90c690bd969c 2023-04-25 02:54:52,760 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=789 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp/B/53dfa555a4a74f9f839d959aae43382a 2023-04-25 02:54:52,769 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=789 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp/C/a53a6bec412a44d2aaeffb999e63b6a3 2023-04-25 02:54:52,773 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp/A/c46e0fe7a3be483e838d90c690bd969c as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A/c46e0fe7a3be483e838d90c690bd969c 2023-04-25 02:54:52,777 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A/c46e0fe7a3be483e838d90c690bd969c, entries=150, sequenceid=789, filesize=11.9 K 2023-04-25 02:54:52,778 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp/B/53dfa555a4a74f9f839d959aae43382a as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B/53dfa555a4a74f9f839d959aae43382a 2023-04-25 02:54:52,781 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B/53dfa555a4a74f9f839d959aae43382a, entries=150, sequenceid=789, filesize=11.9 K 2023-04-25 02:54:52,782 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp/C/a53a6bec412a44d2aaeffb999e63b6a3 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C/a53a6bec412a44d2aaeffb999e63b6a3 2023-04-25 02:54:52,785 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C/a53a6bec412a44d2aaeffb999e63b6a3, entries=150, sequenceid=789, filesize=11.9 K 2023-04-25 02:54:52,786 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 40cf532a35b33bde962f4f28e36e887b in 445ms, sequenceid=789, compaction requested=false 2023-04-25 02:54:52,786 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 40cf532a35b33bde962f4f28e36e887b: 2023-04-25 02:54:52,786 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. 2023-04-25 02:54:52,787 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:54:52,787 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:54:52,787 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:52,787 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:54:52,787 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:54:52,788 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:52,788 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:52,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:52,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:52,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:52,789 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:52,789 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:54:52,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:52,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:52,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:52,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:52,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:52,790 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:54:52,790 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4d3b7699[Count = 0] remaining members to acquire global barrier 2023-04-25 02:54:52,790 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:54:52,790 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:52,791 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:52,791 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:52,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:52,792 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:54:52,792 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:54:52,792 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:54:52,792 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:52,792 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:54:52,793 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:52,793 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:54:52,793 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:52,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:52,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:52,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:52,793 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:54:52,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:52,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:52,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:52,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:52,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:52,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:52,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:52,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:54:52,796 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:54:52,796 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:54:52,796 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:54:52,796 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:54:52,796 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:54:52,796 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:54:52,798 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:52,798 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:52,798 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:52,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:52,798 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:52,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:52,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:52,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:52,799 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:52,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:52,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:52,799 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:52,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:52,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:52,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:52,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:52,800 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:52,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:52,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:52,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:52,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:52,803 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:52,803 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:52,803 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:52,803 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:52,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:52,803 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:54:52,803 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:54:52,803 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:52,803 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:54:52,803 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:52,803 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:52,803 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:52,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:52,803 DEBUG [Thread-1512] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:54:52,804 DEBUG [Thread-1512] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:54:52,804 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:52,804 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:52,804 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:52,804 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:52,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(9158): Flush requested on 40cf532a35b33bde962f4f28e36e887b 2023-04-25 02:54:52,820 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 40cf532a35b33bde962f4f28e36e887b 3/3 column families, dataSize=140.89 KB heapSize=405.61 KB 2023-04-25 02:54:52,820 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 40cf532a35b33bde962f4f28e36e887b, store=A 2023-04-25 02:54:52,820 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-25 02:54:52,820 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 40cf532a35b33bde962f4f28e36e887b, store=B 2023-04-25 02:54:52,820 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-25 02:54:52,820 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 40cf532a35b33bde962f4f28e36e887b, store=C 2023-04-25 02:54:52,820 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-25 02:54:52,832 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:52,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 254 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59160 deadline: 1682391352828, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:52,833 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:52,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 305 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59158 deadline: 1682391352831, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:52,835 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:52,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 226 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59214 deadline: 1682391352832, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:52,837 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=828 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp/A/fe041106d1da491890c6251e54d49bf7 2023-04-25 02:54:52,838 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:52,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59216 deadline: 1682391352836, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:52,840 DEBUG [Thread-1502] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4144 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b., hostname=jenkins-hbase4.apache.org,46697,1682391220023, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor28.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-25 02:54:52,847 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:52,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59186 deadline: 1682391352845, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:52,849 DEBUG [Thread-1504] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4150 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b., hostname=jenkins-hbase4.apache.org,46697,1682391220023, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor28.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-25 02:54:52,849 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=828 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp/B/a92547c229664cbf8f7ccd238769a2c6 2023-04-25 02:54:52,904 DEBUG [Thread-1512] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:54:52,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:54:52,906 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:54:52,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:54:52,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:54:52,907 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:54:52,907 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:52,907 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:54:52,907 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:54:52,908 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:52,908 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:52,909 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:52,909 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:52,909 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:52,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:52,909 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:52,909 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:54:52,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:52,909 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:52,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:54:52,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:52,910 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:54:52,910 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:54:52,910 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:54:52,910 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:54:52,910 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:52,910 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:54:52,910 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:54:52,910 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:54:52,910 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. 2023-04-25 02:54:52,910 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. started... 2023-04-25 02:54:52,911 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. as already flushing 2023-04-25 02:54:52,911 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. 2023-04-25 02:54:52,911 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:54:52,911 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:54:52,911 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:52,911 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:54:52,911 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:54:52,912 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:52,912 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:52,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:52,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:52,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:52,913 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:52,913 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:54:52,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:52,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:52,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:52,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:52,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:52,914 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:54:52,915 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6d35e39f[Count = 0] remaining members to acquire global barrier 2023-04-25 02:54:52,915 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:54:52,915 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:52,917 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:52,917 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:52,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:52,917 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:54:52,917 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:54:52,917 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:54:52,917 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:52,917 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:54:52,918 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:52,918 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:54:52,918 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:52,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:52,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:52,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:52,918 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:54:52,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:52,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:52,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:52,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:52,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:52,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:52,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:52,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:54:52,921 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:54:52,921 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:54:52,921 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:54:52,921 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:54:52,921 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:54:52,921 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:54:52,922 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:52,922 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:52,922 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:52,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:52,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:52,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:52,922 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:52,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:52,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:52,923 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:52,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:52,923 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:52,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:52,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:52,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:52,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:52,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:52,927 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:52,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:52,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:52,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:52,929 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:52,929 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:52,929 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:52,929 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:52,929 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:54:52,929 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:54:52,929 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:52,929 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:52,929 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:52,930 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:52,929 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:52,929 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:52,930 DEBUG [Thread-1512] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:54:52,929 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:52,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:52,930 DEBUG [Thread-1512] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:54:52,930 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:52,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:52,935 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:52,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 256 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59160 deadline: 1682391352933, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:52,936 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:52,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 307 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59158 deadline: 1682391352934, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:52,937 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:52,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 228 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59214 deadline: 1682391352935, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:53,030 DEBUG [Thread-1512] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:54:53,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:54:53,032 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:54:53,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:54:53,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:54:53,033 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:54:53,033 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:53,033 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:54:53,033 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:54:53,034 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:53,034 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:53,036 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:53,036 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:53,036 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:53,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:53,036 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:53,036 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:54:53,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:53,037 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:53,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:54:53,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:53,037 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:54:53,037 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:54:53,037 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:54:53,038 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:54:53,038 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:53,038 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:54:53,038 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:54:53,038 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:54:53,038 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. 2023-04-25 02:54:53,038 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. started... 2023-04-25 02:54:53,038 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. as already flushing 2023-04-25 02:54:53,038 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. 2023-04-25 02:54:53,038 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:54:53,038 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:54:53,038 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:53,038 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:54:53,038 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:54:53,040 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:53,040 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:53,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:53,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:53,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:53,040 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:53,040 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:54:53,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:53,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:53,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:53,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:53,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:53,042 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:54:53,042 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@22a4dd61[Count = 0] remaining members to acquire global barrier 2023-04-25 02:54:53,042 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:54:53,042 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:53,043 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:53,043 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:53,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:53,043 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:54:53,043 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:54:53,043 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:54:53,043 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:53,043 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:54:53,044 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:53,044 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:54:53,044 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:53,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:53,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:53,044 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:54:53,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:53,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:53,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:53,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:53,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:53,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:53,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:53,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:53,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:54:53,046 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:54:53,046 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:54:53,046 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:54:53,046 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:54:53,046 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:54:53,047 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:54:53,049 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:53,049 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:53,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:53,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:53,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:53,049 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:53,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:53,049 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:53,050 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:53,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:53,050 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:53,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:53,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:53,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:53,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:53,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:53,051 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:53,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:53,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:53,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:53,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:53,053 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:53,053 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:53,053 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:53,053 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:53,053 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:54:53,053 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:53,053 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:53,053 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:53,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:53,053 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:54:53,053 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:53,053 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:54:53,053 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:53,054 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:53,054 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:53,054 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:53,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:53,054 DEBUG [Thread-1512] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:54:53,054 DEBUG [Thread-1512] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:54:53,139 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:53,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 309 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59158 deadline: 1682391353137, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:53,139 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:53,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 258 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59160 deadline: 1682391353137, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:53,139 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:53,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 230 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59214 deadline: 1682391353138, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:53,154 DEBUG [Thread-1512] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:54:53,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:54:53,156 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:54:53,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:54:53,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:54:53,157 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:54:53,157 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:53,157 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:54:53,157 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:54:53,158 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:53,158 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:53,159 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:53,159 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:53,159 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:53,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:53,159 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:53,159 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:54:53,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:53,160 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:53,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:54:53,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:53,160 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:54:53,160 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:54:53,160 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:54:53,160 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:54:53,160 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:53,160 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:54:53,161 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:54:53,161 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:54:53,161 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. 2023-04-25 02:54:53,161 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. started... 2023-04-25 02:54:53,161 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. as already flushing 2023-04-25 02:54:53,161 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. 2023-04-25 02:54:53,161 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:54:53,161 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:54:53,161 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:53,161 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:54:53,161 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:54:53,169 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:53,169 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:53,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:53,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:53,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:53,170 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:53,170 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:54:53,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:53,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:53,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:53,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:53,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:53,171 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:54:53,171 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@71a4037[Count = 0] remaining members to acquire global barrier 2023-04-25 02:54:53,171 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:54:53,172 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:53,173 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:53,173 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:53,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:53,173 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:54:53,173 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:54:53,173 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:54:53,173 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:53,173 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:54:53,174 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:53,174 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:54:53,174 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:53,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:53,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:53,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:53,174 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:54:53,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:53,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:53,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:53,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:53,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:53,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:53,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:53,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:54:53,177 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:54:53,177 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:54:53,177 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:54:53,177 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:54:53,177 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:54:53,177 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:54:53,178 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:53,178 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:53,178 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:53,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:53,178 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:53,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:53,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:53,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:53,178 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:53,178 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:53,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:53,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:53,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:53,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:53,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:53,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:53,180 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:53,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:53,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:53,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:53,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:53,186 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:53,186 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:53,186 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:53,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:53,186 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:53,186 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:54:53,186 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:54:53,186 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:54:53,186 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:53,186 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:53,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:53,186 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:53,186 DEBUG [Thread-1512] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:54:53,186 DEBUG [Thread-1512] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:54:53,186 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:53,186 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:53,186 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:53,186 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:53,186 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:53,259 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=828 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp/C/b06f0f1632084ffda1167091ff512f6e 2023-04-25 02:54:53,263 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp/A/fe041106d1da491890c6251e54d49bf7 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A/fe041106d1da491890c6251e54d49bf7 2023-04-25 02:54:53,267 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A/fe041106d1da491890c6251e54d49bf7, entries=150, sequenceid=828, filesize=11.9 K 2023-04-25 02:54:53,268 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp/B/a92547c229664cbf8f7ccd238769a2c6 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B/a92547c229664cbf8f7ccd238769a2c6 2023-04-25 02:54:53,272 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B/a92547c229664cbf8f7ccd238769a2c6, entries=150, sequenceid=828, filesize=11.9 K 2023-04-25 02:54:53,273 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp/C/b06f0f1632084ffda1167091ff512f6e as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C/b06f0f1632084ffda1167091ff512f6e 2023-04-25 02:54:53,276 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C/b06f0f1632084ffda1167091ff512f6e, entries=150, sequenceid=828, filesize=11.9 K 2023-04-25 02:54:53,277 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~405.56 KB/415296, currentSize=40.25 KB/41220 for 40cf532a35b33bde962f4f28e36e887b in 457ms, sequenceid=828, compaction requested=true 2023-04-25 02:54:53,277 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 40cf532a35b33bde962f4f28e36e887b: 2023-04-25 02:54:53,277 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 02:54:53,277 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-25 02:54:53,277 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 02:54:53,277 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-25 02:54:53,277 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 02:54:53,278 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37308 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-25 02:54:53,278 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37308 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-25 02:54:53,278 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HStore(1912): 40cf532a35b33bde962f4f28e36e887b/A is initiating minor compaction (all files) 2023-04-25 02:54:53,278 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HStore(1912): 40cf532a35b33bde962f4f28e36e887b/B is initiating minor compaction (all files) 2023-04-25 02:54:53,278 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 40cf532a35b33bde962f4f28e36e887b/A in TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. 2023-04-25 02:54:53,278 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 40cf532a35b33bde962f4f28e36e887b/B in TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. 2023-04-25 02:54:53,278 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A/6d781aadadbf44b98ee0c11862c78b54, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A/c46e0fe7a3be483e838d90c690bd969c, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A/fe041106d1da491890c6251e54d49bf7] into tmpdir=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp, totalSize=36.4 K 2023-04-25 02:54:53,278 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B/76ce613289ae4db386e81e6e16d7c4ce, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B/53dfa555a4a74f9f839d959aae43382a, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B/a92547c229664cbf8f7ccd238769a2c6] into tmpdir=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp, totalSize=36.4 K 2023-04-25 02:54:53,279 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting 6d781aadadbf44b98ee0c11862c78b54, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=777, earliestPutTs=1682391290695 2023-04-25 02:54:53,279 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting 76ce613289ae4db386e81e6e16d7c4ce, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=777, earliestPutTs=1682391290695 2023-04-25 02:54:53,279 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting c46e0fe7a3be483e838d90c690bd969c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=789, earliestPutTs=1682391291775 2023-04-25 02:54:53,279 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting 53dfa555a4a74f9f839d959aae43382a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=789, earliestPutTs=1682391291775 2023-04-25 02:54:53,279 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting fe041106d1da491890c6251e54d49bf7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=828, earliestPutTs=1682391292488 2023-04-25 02:54:53,279 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting a92547c229664cbf8f7ccd238769a2c6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=828, earliestPutTs=1682391292488 2023-04-25 02:54:53,285 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] throttle.PressureAwareThroughputController(145): 40cf532a35b33bde962f4f28e36e887b#A#compaction#258 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 53.85 MB/second 2023-04-25 02:54:53,286 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] throttle.PressureAwareThroughputController(145): 40cf532a35b33bde962f4f28e36e887b#B#compaction#259 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-25 02:54:53,287 DEBUG [Thread-1512] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:54:53,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:54:53,289 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:54:53,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:54:53,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:54:53,292 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:54:53,292 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:53,292 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:54:53,292 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:54:53,293 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:53,293 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:53,294 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:53,294 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:53,294 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:53,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:53,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:53,294 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:53,294 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:54:53,295 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:53,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:54:53,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:53,295 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:54:53,295 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:54:53,295 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:54:53,296 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:54:53,296 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:53,296 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:54:53,296 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:54:53,296 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:54:53,296 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. 2023-04-25 02:54:53,296 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. started... 2023-04-25 02:54:53,296 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 40cf532a35b33bde962f4f28e36e887b 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-04-25 02:54:53,296 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 40cf532a35b33bde962f4f28e36e887b, store=A 2023-04-25 02:54:53,296 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:54:53,296 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 40cf532a35b33bde962f4f28e36e887b, store=B 2023-04-25 02:54:53,296 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:54:53,296 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 40cf532a35b33bde962f4f28e36e887b, store=C 2023-04-25 02:54:53,297 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:54:53,299 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp/B/6ab5c5473c38478397548c94668ae5ea as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B/6ab5c5473c38478397548c94668ae5ea 2023-04-25 02:54:53,304 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=837 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp/A/9216d1ac67964930bc892bf6cdd5da91 2023-04-25 02:54:53,304 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 40cf532a35b33bde962f4f28e36e887b/B of 40cf532a35b33bde962f4f28e36e887b into 6ab5c5473c38478397548c94668ae5ea(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 02:54:53,304 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 40cf532a35b33bde962f4f28e36e887b: 2023-04-25 02:54:53,304 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b., storeName=40cf532a35b33bde962f4f28e36e887b/B, priority=13, startTime=1682391293277; duration=0sec 2023-04-25 02:54:53,304 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 02:54:53,304 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-25 02:54:53,305 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37308 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-25 02:54:53,306 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HStore(1912): 40cf532a35b33bde962f4f28e36e887b/C is initiating minor compaction (all files) 2023-04-25 02:54:53,306 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 40cf532a35b33bde962f4f28e36e887b/C in TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. 2023-04-25 02:54:53,306 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C/7c2a1f49b23a4fe08ab8c2567609d5aa, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C/a53a6bec412a44d2aaeffb999e63b6a3, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C/b06f0f1632084ffda1167091ff512f6e] into tmpdir=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp, totalSize=36.4 K 2023-04-25 02:54:53,306 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting 7c2a1f49b23a4fe08ab8c2567609d5aa, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=777, earliestPutTs=1682391290695 2023-04-25 02:54:53,307 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting a53a6bec412a44d2aaeffb999e63b6a3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=789, earliestPutTs=1682391291775 2023-04-25 02:54:53,307 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting b06f0f1632084ffda1167091ff512f6e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=828, earliestPutTs=1682391292488 2023-04-25 02:54:53,314 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] throttle.PressureAwareThroughputController(145): 40cf532a35b33bde962f4f28e36e887b#C#compaction#262 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-25 02:54:53,319 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=837 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp/B/c28fe9c09f8d40ec8e2ad2d0ee6e1b23 2023-04-25 02:54:53,325 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp/C/ca9b6e0a629f4db186e5b8a47293b8fe as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C/ca9b6e0a629f4db186e5b8a47293b8fe 2023-04-25 02:54:53,331 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=837 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp/C/2388ec51e2ea4ddd804098b60f33c29c 2023-04-25 02:54:53,332 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 40cf532a35b33bde962f4f28e36e887b/C of 40cf532a35b33bde962f4f28e36e887b into ca9b6e0a629f4db186e5b8a47293b8fe(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 02:54:53,332 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 40cf532a35b33bde962f4f28e36e887b: 2023-04-25 02:54:53,332 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b., storeName=40cf532a35b33bde962f4f28e36e887b/C, priority=13, startTime=1682391293277; duration=0sec 2023-04-25 02:54:53,332 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 02:54:53,335 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp/A/9216d1ac67964930bc892bf6cdd5da91 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A/9216d1ac67964930bc892bf6cdd5da91 2023-04-25 02:54:53,339 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A/9216d1ac67964930bc892bf6cdd5da91, entries=150, sequenceid=837, filesize=11.9 K 2023-04-25 02:54:53,340 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp/B/c28fe9c09f8d40ec8e2ad2d0ee6e1b23 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B/c28fe9c09f8d40ec8e2ad2d0ee6e1b23 2023-04-25 02:54:53,343 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B/c28fe9c09f8d40ec8e2ad2d0ee6e1b23, entries=150, sequenceid=837, filesize=11.9 K 2023-04-25 02:54:53,344 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp/C/2388ec51e2ea4ddd804098b60f33c29c as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C/2388ec51e2ea4ddd804098b60f33c29c 2023-04-25 02:54:53,347 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C/2388ec51e2ea4ddd804098b60f33c29c, entries=150, sequenceid=837, filesize=11.9 K 2023-04-25 02:54:53,348 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=0 B/0 for 40cf532a35b33bde962f4f28e36e887b in 52ms, sequenceid=837, compaction requested=false 2023-04-25 02:54:53,348 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 40cf532a35b33bde962f4f28e36e887b: 2023-04-25 02:54:53,348 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. 2023-04-25 02:54:53,348 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:54:53,348 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:54:53,348 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:53,348 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:54:53,348 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:54:53,349 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:53,349 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:53,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:53,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:53,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:53,350 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:53,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:53,350 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:54:53,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:53,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:53,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:53,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:53,352 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:54:53,352 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@110361f3[Count = 0] remaining members to acquire global barrier 2023-04-25 02:54:53,352 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:54:53,352 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:53,353 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:53,353 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:53,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:53,353 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:54:53,353 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:54:53,353 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:54:53,354 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:53,354 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:54:53,355 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:53,355 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:54:53,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:53,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:53,355 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:53,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:53,356 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:54:53,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:53,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:53,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:53,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:53,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:53,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:53,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:53,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:54:53,358 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:54:53,358 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:54:53,358 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:54:53,358 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:54:53,358 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:54:53,358 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:54:53,359 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:53,359 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:53,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:53,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:53,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:53,360 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:53,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:53,360 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:53,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:53,360 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:53,360 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:53,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:53,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:53,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:53,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:53,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:53,361 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:53,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:53,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:53,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:53,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:53,363 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:53,363 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:53,363 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:53,363 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:53,363 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:54:53,364 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:54:53,364 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:54:53,364 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:53,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:53,363 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:53,363 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:53,364 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:53,364 DEBUG [Thread-1512] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:54:53,364 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:53,364 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:53,364 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:53,364 DEBUG [Thread-1512] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:54:53,364 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:53,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:53,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(9158): Flush requested on 40cf532a35b33bde962f4f28e36e887b 2023-04-25 02:54:53,446 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 40cf532a35b33bde962f4f28e36e887b 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-25 02:54:53,447 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 40cf532a35b33bde962f4f28e36e887b, store=A 2023-04-25 02:54:53,447 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:54:53,447 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 40cf532a35b33bde962f4f28e36e887b, store=B 2023-04-25 02:54:53,447 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:54:53,447 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 40cf532a35b33bde962f4f28e36e887b, store=C 2023-04-25 02:54:53,447 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:54:53,454 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=850 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp/A/fa3a7b9844c34827a46099a848c2fb05 2023-04-25 02:54:53,464 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=850 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp/B/5f9e6a0a308344a59b01681fd7734911 2023-04-25 02:54:53,464 DEBUG [Thread-1512] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:54:53,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:54:53,467 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:54:53,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:54:53,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:54:53,468 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:54:53,468 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:53,468 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:54:53,468 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:54:53,469 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:53,469 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:53,470 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:53,470 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:53,470 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:53,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:53,471 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:53,471 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:54:53,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:53,472 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:53,472 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:53,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 240 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59214 deadline: 1682391353469, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:53,473 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682391293467, max timestamp=1682391293471Num uniques 859; 2023-04-25 02:54:53,473 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-25 02:54:53,473 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682391293467, max timestamp=1682391293471Num uniques 859; 2023-04-25 02:54:53,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:54:53,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:53,473 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:54:53,473 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682391293467, max timestamp=1682391293471Num uniques 859; 2023-04-25 02:54:53,474 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-25 02:54:53,473 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-25 02:54:53,474 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:54:53,474 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:54:53,474 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:54:53,474 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:53,474 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:54:53,474 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:54:53,474 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:54:53,474 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. 2023-04-25 02:54:53,474 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. started... 2023-04-25 02:54:53,474 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. as already flushing 2023-04-25 02:54:53,474 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. 2023-04-25 02:54:53,475 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:54:53,475 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:54:53,475 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:53,475 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:54:53,475 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:54:53,477 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:53,477 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:53,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:53,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:53,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:53,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:53,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:53,478 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:53,478 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:54:53,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:53,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:53,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:53,479 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:54:53,479 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@53685157[Count = 0] remaining members to acquire global barrier 2023-04-25 02:54:53,479 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:54:53,479 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:53,481 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:53,481 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:53,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:53,481 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:54:53,481 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:53,481 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:54:53,481 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:54:53,481 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:54:53,482 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:54:53,482 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:53,482 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:53,482 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:54:53,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:53,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:53,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:53,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:53,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:53,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:53,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:53,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:53,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:53,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:53,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:54:53,485 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:54:53,485 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:54:53,485 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:54:53,485 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:54:53,485 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:54:53,485 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:54:53,486 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:53,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:53,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:53,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:53,486 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:53,486 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:53,486 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:53,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:53,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:53,487 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:53,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:53,487 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:53,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:53,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:53,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:53,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:53,488 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:53,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:53,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:53,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:53,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:53,491 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:53,491 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:53,491 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:53,491 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:53,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:53,491 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:54:53,491 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:53,491 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:54:53,491 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:54:53,491 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:53,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:53,491 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:53,491 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:53,492 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:53,492 DEBUG [Thread-1512] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:54:53,492 DEBUG [Thread-1512] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:54:53,492 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:53,492 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:53,492 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:53,507 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:53,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 327 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59158 deadline: 1682391353503, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:53,507 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:53,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 277 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59160 deadline: 1682391353504, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:53,576 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:53,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 242 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59214 deadline: 1682391353574, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:53,592 DEBUG [Thread-1512] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:54:53,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:54:53,594 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:54:53,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:54:53,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:54:53,595 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:54:53,595 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:53,595 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:54:53,595 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:54:53,595 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:53,596 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:53,597 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:53,597 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:53,597 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:53,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:53,597 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:53,597 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:54:53,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:53,597 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:53,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:54:53,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:53,598 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:54:53,598 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:54:53,598 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:54:53,598 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:54:53,598 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:53,598 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:54:53,598 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:54:53,598 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:54:53,598 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. 2023-04-25 02:54:53,598 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. started... 2023-04-25 02:54:53,598 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. as already flushing 2023-04-25 02:54:53,598 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. 2023-04-25 02:54:53,598 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:54:53,599 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:54:53,599 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:53,599 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:54:53,599 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:54:53,600 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:53,600 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:53,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:53,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:53,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:53,600 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:53,600 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:54:53,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:53,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:53,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:53,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:53,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:53,602 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:54:53,602 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5723c18e[Count = 0] remaining members to acquire global barrier 2023-04-25 02:54:53,602 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:54:53,602 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:53,603 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:53,603 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:53,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:53,603 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:54:53,603 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:54:53,603 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:53,603 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:54:53,603 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:54:53,608 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:53,608 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:54:53,608 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:53,608 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:54:53,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:53,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:53,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:53,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:53,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:53,609 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:53,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 329 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59158 deadline: 1682391353608, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:53,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:53,609 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:53,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 279 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59160 deadline: 1682391353608, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:53,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:53,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:53,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:53,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:53,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:54:53,611 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:54:53,611 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:54:53,611 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:54:53,611 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:54:53,611 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:54:53,611 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:54:53,612 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:53,612 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:53,612 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:53,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:53,612 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:53,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:53,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:53,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:53,613 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:53,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:53,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:53,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:53,613 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:53,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:53,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:53,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:53,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:53,614 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:53,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:53,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:53,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:53,616 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:53,616 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:53,616 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:53,616 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:54:53,616 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:54:53,616 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:53,616 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:53,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:53,616 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:53,616 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:53,616 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:53,617 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:53,617 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:53,617 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:53,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:53,617 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:53,617 DEBUG [Thread-1512] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:54:53,617 DEBUG [Thread-1512] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:54:53,696 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp/A/2e2e8c94bc304019a39d9caa97d8a251 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A/2e2e8c94bc304019a39d9caa97d8a251 2023-04-25 02:54:53,700 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 40cf532a35b33bde962f4f28e36e887b/A of 40cf532a35b33bde962f4f28e36e887b into 2e2e8c94bc304019a39d9caa97d8a251(size=12.7 K), total size for store is 24.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 02:54:53,700 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 40cf532a35b33bde962f4f28e36e887b: 2023-04-25 02:54:53,700 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b., storeName=40cf532a35b33bde962f4f28e36e887b/A, priority=13, startTime=1682391293277; duration=0sec 2023-04-25 02:54:53,701 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 02:54:53,717 DEBUG [Thread-1512] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:54:53,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:54:53,719 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:54:53,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:54:53,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:54:53,720 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:54:53,720 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:53,721 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:54:53,721 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:54:53,721 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:53,721 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:53,722 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:53,722 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:53,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:53,722 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:53,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:53,722 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:53,723 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:54:53,723 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:53,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:54:53,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:53,723 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:54:53,723 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:54:53,723 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:54:53,723 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:54:53,723 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:53,723 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:54:53,724 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:54:53,724 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:54:53,724 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. 2023-04-25 02:54:53,724 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. started... 2023-04-25 02:54:53,724 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. as already flushing 2023-04-25 02:54:53,724 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. 2023-04-25 02:54:53,724 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:54:53,724 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:54:53,724 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:53,724 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:54:53,724 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:54:53,726 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:53,726 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:53,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:53,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:53,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:53,727 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:53,727 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:54:53,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:53,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:53,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:53,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:53,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:53,728 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:54:53,728 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@26d28a5[Count = 0] remaining members to acquire global barrier 2023-04-25 02:54:53,728 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:54:53,728 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:53,730 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:53,730 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:53,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:53,730 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:54:53,730 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:54:53,730 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:54:53,730 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:53,730 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:54:53,732 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:53,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:53,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:53,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:53,736 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:54:53,736 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:53,736 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:54:53,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:53,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:53,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:53,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:53,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:53,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:53,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:53,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:54:53,742 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:54:53,742 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:54:53,742 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:54:53,742 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:54:53,743 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:54:53,743 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:54:53,744 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:53,744 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:53,744 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:53,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:53,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:53,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:53,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:53,744 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:53,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:53,744 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:53,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:53,744 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:53,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:53,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:53,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:53,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:53,746 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:53,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:53,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:53,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:53,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:53,749 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:53,749 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:53,749 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:53,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:53,749 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:53,749 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:53,749 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:53,749 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:54:53,749 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:54:53,749 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:54:53,749 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:53,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:53,749 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:53,749 DEBUG [Thread-1512] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:54:53,749 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:53,749 DEBUG [Thread-1512] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:54:53,749 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:53,749 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:53,749 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:53,778 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:53,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 244 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59214 deadline: 1682391353777, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:53,812 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:53,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 331 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59158 deadline: 1682391353811, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:53,813 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:53,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 281 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59160 deadline: 1682391353811, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:53,850 DEBUG [Thread-1512] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:54:53,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:54:53,852 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:54:53,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:54:53,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:54:53,854 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:54:53,854 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:53,854 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:54:53,854 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:54:53,854 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:53,854 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:53,855 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:53,856 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:53,856 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:53,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:53,856 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:53,856 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:54:53,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:53,856 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:53,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:54:53,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:53,857 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:54:53,857 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:54:53,857 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:54:53,857 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:54:53,857 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:53,857 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:54:53,857 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:54:53,857 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:54:53,857 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. 2023-04-25 02:54:53,857 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. started... 2023-04-25 02:54:53,858 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. as already flushing 2023-04-25 02:54:53,858 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. 2023-04-25 02:54:53,858 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:54:53,858 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:54:53,858 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:53,858 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:54:53,858 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:54:53,859 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:53,859 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:53,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:53,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:53,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:53,859 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:53,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:53,859 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:54:53,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:53,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:53,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:53,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:53,861 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:54:53,861 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@86bdde2[Count = 0] remaining members to acquire global barrier 2023-04-25 02:54:53,861 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:54:53,861 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:53,862 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:53,862 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:53,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:53,862 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:54:53,862 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:54:53,862 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:54:53,862 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:53,862 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:54:53,864 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:53,864 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:54:53,864 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:53,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:53,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:53,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:53,864 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:54:53,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:53,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:53,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:53,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:53,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:53,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:53,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:53,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:54:53,867 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:54:53,867 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:54:53,867 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:54:53,867 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:54:53,867 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:54:53,867 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:54:53,868 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:53,868 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:53,868 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:53,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:53,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:53,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:53,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:53,868 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:53,868 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:53,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:53,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:53,869 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:53,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:53,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:53,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:53,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:53,870 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:53,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:53,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:53,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:53,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:53,872 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:53,872 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:53,872 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:53,872 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:53,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:53,872 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:53,872 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:53,872 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:54:53,873 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:54:53,873 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:53,873 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:53,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:53,873 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:53,873 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:53,873 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:53,873 DEBUG [Thread-1512] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:54:53,873 DEBUG [Thread-1512] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:54:53,873 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:53,876 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=850 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp/C/37f186c853bb4d1bbc4c5996b10e31c2 2023-04-25 02:54:53,880 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp/A/fa3a7b9844c34827a46099a848c2fb05 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A/fa3a7b9844c34827a46099a848c2fb05 2023-04-25 02:54:53,884 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A/fa3a7b9844c34827a46099a848c2fb05, entries=100, sequenceid=850, filesize=9.5 K 2023-04-25 02:54:53,885 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp/B/5f9e6a0a308344a59b01681fd7734911 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B/5f9e6a0a308344a59b01681fd7734911 2023-04-25 02:54:53,888 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B/5f9e6a0a308344a59b01681fd7734911, entries=100, sequenceid=850, filesize=9.5 K 2023-04-25 02:54:53,889 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp/C/37f186c853bb4d1bbc4c5996b10e31c2 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C/37f186c853bb4d1bbc4c5996b10e31c2 2023-04-25 02:54:53,892 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C/37f186c853bb4d1bbc4c5996b10e31c2, entries=100, sequenceid=850, filesize=9.5 K 2023-04-25 02:54:53,893 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 40cf532a35b33bde962f4f28e36e887b in 447ms, sequenceid=850, compaction requested=true 2023-04-25 02:54:53,893 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 40cf532a35b33bde962f4f28e36e887b: 2023-04-25 02:54:53,893 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 02:54:53,893 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-25 02:54:53,893 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-25 02:54:53,893 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 02:54:53,893 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 02:54:53,894 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34966 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-25 02:54:53,894 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HStore(1912): 40cf532a35b33bde962f4f28e36e887b/A is initiating minor compaction (all files) 2023-04-25 02:54:53,894 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 40cf532a35b33bde962f4f28e36e887b/A in TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. 2023-04-25 02:54:53,895 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A/2e2e8c94bc304019a39d9caa97d8a251, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A/9216d1ac67964930bc892bf6cdd5da91, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A/fa3a7b9844c34827a46099a848c2fb05] into tmpdir=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp, totalSize=34.1 K 2023-04-25 02:54:53,895 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34966 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-25 02:54:53,895 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting 2e2e8c94bc304019a39d9caa97d8a251, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=828, earliestPutTs=1682391292488 2023-04-25 02:54:53,895 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HStore(1912): 40cf532a35b33bde962f4f28e36e887b/B is initiating minor compaction (all files) 2023-04-25 02:54:53,895 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 40cf532a35b33bde962f4f28e36e887b/B in TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. 2023-04-25 02:54:53,895 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B/6ab5c5473c38478397548c94668ae5ea, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B/c28fe9c09f8d40ec8e2ad2d0ee6e1b23, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B/5f9e6a0a308344a59b01681fd7734911] into tmpdir=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp, totalSize=34.1 K 2023-04-25 02:54:53,896 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting 9216d1ac67964930bc892bf6cdd5da91, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=837, earliestPutTs=1682391292825 2023-04-25 02:54:53,896 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting 6ab5c5473c38478397548c94668ae5ea, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=828, earliestPutTs=1682391292488 2023-04-25 02:54:53,896 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting fa3a7b9844c34827a46099a848c2fb05, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=850, earliestPutTs=1682391293444 2023-04-25 02:54:53,896 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting c28fe9c09f8d40ec8e2ad2d0ee6e1b23, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=837, earliestPutTs=1682391292825 2023-04-25 02:54:53,896 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting 5f9e6a0a308344a59b01681fd7734911, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=850, earliestPutTs=1682391293444 2023-04-25 02:54:53,906 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] throttle.PressureAwareThroughputController(145): 40cf532a35b33bde962f4f28e36e887b#A#compaction#267 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 53.85 MB/second 2023-04-25 02:54:53,907 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] throttle.PressureAwareThroughputController(145): 40cf532a35b33bde962f4f28e36e887b#B#compaction#268 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-25 02:54:53,973 DEBUG [Thread-1512] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:54:53,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:54:53,975 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:54:53,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:54:53,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:54:53,977 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:54:53,977 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:53,977 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:54:53,977 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:54:53,977 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:53,977 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:53,978 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:53,978 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:53,978 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:53,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:53,978 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:53,979 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:54:53,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:53,979 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:53,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:54:53,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:53,979 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:54:53,980 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:54:53,980 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:54:53,980 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:54:53,980 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:53,980 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:54:53,980 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:54:53,980 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:54:53,980 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. 2023-04-25 02:54:53,980 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. started... 2023-04-25 02:54:53,980 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 40cf532a35b33bde962f4f28e36e887b 3/3 column families, dataSize=127.47 KB heapSize=365.06 KB 2023-04-25 02:54:53,980 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 40cf532a35b33bde962f4f28e36e887b, store=A 2023-04-25 02:54:53,981 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 02:54:53,981 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 40cf532a35b33bde962f4f28e36e887b, store=B 2023-04-25 02:54:53,981 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 02:54:53,981 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 40cf532a35b33bde962f4f28e36e887b, store=C 2023-04-25 02:54:53,981 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 02:54:53,988 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=887 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp/A/5dd3a2cc21f240f0b9edd04b30b6f144 2023-04-25 02:54:53,997 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=887 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp/B/8d0bbdd1546346ecba47c226ac7e9b7d 2023-04-25 02:54:54,081 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(9158): Flush requested on 40cf532a35b33bde962f4f28e36e887b 2023-04-25 02:54:54,081 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. as already flushing 2023-04-25 02:54:54,104 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:54,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 254 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59214 deadline: 1682391354101, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:54,117 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:54,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 283 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59160 deadline: 1682391354113, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:54,117 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:54,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 333 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59158 deadline: 1682391354115, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:54,206 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:54,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 256 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59214 deadline: 1682391354205, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:54,318 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp/A/6aa5474d0e4b485bb50d172e4a656210 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A/6aa5474d0e4b485bb50d172e4a656210 2023-04-25 02:54:54,323 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 40cf532a35b33bde962f4f28e36e887b/A of 40cf532a35b33bde962f4f28e36e887b into 6aa5474d0e4b485bb50d172e4a656210(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 02:54:54,323 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 40cf532a35b33bde962f4f28e36e887b: 2023-04-25 02:54:54,323 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b., storeName=40cf532a35b33bde962f4f28e36e887b/A, priority=13, startTime=1682391293893; duration=0sec 2023-04-25 02:54:54,323 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 02:54:54,323 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-25 02:54:54,325 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34966 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-25 02:54:54,325 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HStore(1912): 40cf532a35b33bde962f4f28e36e887b/C is initiating minor compaction (all files) 2023-04-25 02:54:54,325 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 40cf532a35b33bde962f4f28e36e887b/C in TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. 2023-04-25 02:54:54,325 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C/ca9b6e0a629f4db186e5b8a47293b8fe, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C/2388ec51e2ea4ddd804098b60f33c29c, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C/37f186c853bb4d1bbc4c5996b10e31c2] into tmpdir=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp, totalSize=34.1 K 2023-04-25 02:54:54,325 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting ca9b6e0a629f4db186e5b8a47293b8fe, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=828, earliestPutTs=1682391292488 2023-04-25 02:54:54,326 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting 2388ec51e2ea4ddd804098b60f33c29c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=837, earliestPutTs=1682391292825 2023-04-25 02:54:54,326 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting 37f186c853bb4d1bbc4c5996b10e31c2, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=850, earliestPutTs=1682391293444 2023-04-25 02:54:54,326 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp/B/5041509541624492accd344faa62e0d3 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B/5041509541624492accd344faa62e0d3 2023-04-25 02:54:54,331 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 40cf532a35b33bde962f4f28e36e887b/B of 40cf532a35b33bde962f4f28e36e887b into 5041509541624492accd344faa62e0d3(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 02:54:54,331 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 40cf532a35b33bde962f4f28e36e887b: 2023-04-25 02:54:54,331 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b., storeName=40cf532a35b33bde962f4f28e36e887b/B, priority=13, startTime=1682391293893; duration=0sec 2023-04-25 02:54:54,331 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 02:54:54,333 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] throttle.PressureAwareThroughputController(145): 40cf532a35b33bde962f4f28e36e887b#C#compaction#272 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-25 02:54:54,347 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp/C/f78bd380cc2c4215b6527e71f3259150 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C/f78bd380cc2c4215b6527e71f3259150 2023-04-25 02:54:54,353 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 40cf532a35b33bde962f4f28e36e887b/C of 40cf532a35b33bde962f4f28e36e887b into f78bd380cc2c4215b6527e71f3259150(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 02:54:54,353 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 40cf532a35b33bde962f4f28e36e887b: 2023-04-25 02:54:54,353 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b., storeName=40cf532a35b33bde962f4f28e36e887b/C, priority=13, startTime=1682391293893; duration=0sec 2023-04-25 02:54:54,354 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 02:54:54,407 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=887 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp/C/4a82ea8131454324850d359e769b09f5 2023-04-25 02:54:54,409 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:54,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 258 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59214 deadline: 1682391354408, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:54,411 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp/A/5dd3a2cc21f240f0b9edd04b30b6f144 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A/5dd3a2cc21f240f0b9edd04b30b6f144 2023-04-25 02:54:54,415 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A/5dd3a2cc21f240f0b9edd04b30b6f144, entries=150, sequenceid=887, filesize=11.9 K 2023-04-25 02:54:54,416 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp/B/8d0bbdd1546346ecba47c226ac7e9b7d as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B/8d0bbdd1546346ecba47c226ac7e9b7d 2023-04-25 02:54:54,420 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B/8d0bbdd1546346ecba47c226ac7e9b7d, entries=150, sequenceid=887, filesize=11.9 K 2023-04-25 02:54:54,420 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp/C/4a82ea8131454324850d359e769b09f5 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C/4a82ea8131454324850d359e769b09f5 2023-04-25 02:54:54,425 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C/4a82ea8131454324850d359e769b09f5, entries=150, sequenceid=887, filesize=11.9 K 2023-04-25 02:54:54,425 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~365.02 KB/373776, currentSize=53.67 KB/54960 for 40cf532a35b33bde962f4f28e36e887b in 445ms, sequenceid=887, compaction requested=false 2023-04-25 02:54:54,425 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 40cf532a35b33bde962f4f28e36e887b: 2023-04-25 02:54:54,425 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. 2023-04-25 02:54:54,425 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:54:54,426 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:54:54,426 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:54,426 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:54:54,426 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:54:54,428 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:54,428 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:54,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:54,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:54,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:54,429 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:54,429 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:54:54,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:54,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:54,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:54,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:54,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:54,431 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:54:54,431 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1998d751[Count = 0] remaining members to acquire global barrier 2023-04-25 02:54:54,431 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:54:54,431 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:54,432 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:54,432 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:54,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:54,432 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:54:54,432 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:54:54,432 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:54:54,432 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:54,432 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:54:54,433 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:54,433 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:54:54,433 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:54,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:54,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:54,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:54,433 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:54:54,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:54,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:54,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:54,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:54,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:54,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:54,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:54,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:54:54,436 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:54:54,436 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:54:54,436 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:54:54,436 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:54:54,436 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:54:54,436 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:54:54,441 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:54,441 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:54,441 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:54,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:54,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:54,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:54,441 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:54,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:54,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:54,441 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:54,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:54,441 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:54,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:54,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:54,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:54,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:54,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:54,443 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:54,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:54,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:54,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:54,445 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:54,445 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:54,446 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:54,446 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:54:54,446 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:54:54,446 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:54,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:54,446 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:54,446 DEBUG [Thread-1512] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:54:54,446 DEBUG [Thread-1512] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:54:54,446 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:54,446 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:54,446 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:54,446 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:54,446 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:54,446 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:54,447 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:54,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:54,546 DEBUG [Thread-1512] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:54:54,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:54:54,548 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:54:54,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:54:54,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:54:54,550 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:54:54,550 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:54,550 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:54:54,550 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:54:54,550 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:54,550 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:54,551 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:54,551 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:54,551 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:54,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:54,552 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:54,552 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:54:54,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:54,552 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:54,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:54:54,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:54,552 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:54:54,553 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:54:54,553 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:54:54,553 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:54:54,553 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:54,553 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:54:54,553 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:54:54,553 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:54:54,553 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. 2023-04-25 02:54:54,553 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. started... 2023-04-25 02:54:54,553 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 40cf532a35b33bde962f4f28e36e887b 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-25 02:54:54,554 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 40cf532a35b33bde962f4f28e36e887b, store=A 2023-04-25 02:54:54,554 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:54:54,554 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 40cf532a35b33bde962f4f28e36e887b, store=B 2023-04-25 02:54:54,554 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:54:54,554 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 40cf532a35b33bde962f4f28e36e887b, store=C 2023-04-25 02:54:54,554 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:54:54,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(9158): Flush requested on 40cf532a35b33bde962f4f28e36e887b 2023-04-25 02:54:54,622 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. as already flushing 2023-04-25 02:54:54,651 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682391294639, max timestamp=1682391294650Num uniques 859; 2023-04-25 02:54:54,651 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682391294639, max timestamp=1682391294650Num uniques 859; 2023-04-25 02:54:54,651 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-25 02:54:54,651 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-25 02:54:54,652 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682391294639, max timestamp=1682391294650Num uniques 859; 2023-04-25 02:54:54,652 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-25 02:54:54,681 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:54,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 351 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59158 deadline: 1682391354678, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:54,682 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:54,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 301 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59160 deadline: 1682391354679, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:54,712 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:54,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 260 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59214 deadline: 1682391354711, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:54,784 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:54,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 353 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59158 deadline: 1682391354782, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:54,784 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:54,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 303 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59160 deadline: 1682391354783, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:54,962 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=901 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp/A/febc3a0bd39344d889abb5673b6660fb 2023-04-25 02:54:54,971 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=901 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp/B/9a04fd7bb7224b39b8d90a74d0a53681 2023-04-25 02:54:54,982 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=901 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp/C/e94e9f097d5448a59f5dfa2babd43a9d 2023-04-25 02:54:54,986 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp/A/febc3a0bd39344d889abb5673b6660fb as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A/febc3a0bd39344d889abb5673b6660fb 2023-04-25 02:54:54,987 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:54,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 355 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59158 deadline: 1682391354985, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:54,989 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:54,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 305 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59160 deadline: 1682391354986, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:54,990 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A/febc3a0bd39344d889abb5673b6660fb, entries=150, sequenceid=901, filesize=11.9 K 2023-04-25 02:54:54,990 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp/B/9a04fd7bb7224b39b8d90a74d0a53681 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B/9a04fd7bb7224b39b8d90a74d0a53681 2023-04-25 02:54:54,994 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B/9a04fd7bb7224b39b8d90a74d0a53681, entries=150, sequenceid=901, filesize=11.9 K 2023-04-25 02:54:55,001 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp/C/e94e9f097d5448a59f5dfa2babd43a9d as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C/e94e9f097d5448a59f5dfa2babd43a9d 2023-04-25 02:54:55,007 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C/e94e9f097d5448a59f5dfa2babd43a9d, entries=150, sequenceid=901, filesize=11.9 K 2023-04-25 02:54:55,008 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 40cf532a35b33bde962f4f28e36e887b in 455ms, sequenceid=901, compaction requested=true 2023-04-25 02:54:55,008 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 40cf532a35b33bde962f4f28e36e887b: 2023-04-25 02:54:55,008 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. 2023-04-25 02:54:55,008 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:54:55,008 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:54:55,008 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:55,008 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:54:55,008 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:54:55,009 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:55,009 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:55,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:55,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:55,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:55,010 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:55,010 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:54:55,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:55,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:55,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:55,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:55,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:55,011 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:54:55,011 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@50fe4a5d[Count = 0] remaining members to acquire global barrier 2023-04-25 02:54:55,011 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:54:55,011 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:55,013 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:55,013 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:55,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:55,013 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:54:55,013 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:54:55,013 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:54:55,013 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:55,013 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:54:55,014 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:55,014 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:54:55,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:55,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:55,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:55,014 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:55,015 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:54:55,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:55,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:55,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:55,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:55,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:55,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:55,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:55,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:54:55,027 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:54:55,027 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:54:55,027 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:54:55,027 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:54:55,027 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:54:55,027 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:54:55,028 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:55,028 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:55,028 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:55,028 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:55,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:55,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:55,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:55,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:55,029 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:55,029 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:55,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:55,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:55,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:55,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:55,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:55,030 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:55,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:55,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:55,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:55,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:55,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:55,032 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:55,032 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:55,032 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:55,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:55,032 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:55,032 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:55,032 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:55,032 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:54:55,032 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:54:55,032 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:55,032 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:55,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:55,032 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:55,032 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:55,032 DEBUG [Thread-1512] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:54:55,032 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:55,032 DEBUG [Thread-1512] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:54:55,032 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:55,040 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:54:55,133 DEBUG [Thread-1512] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:54:55,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:54:55,135 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:54:55,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:54:55,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:54:55,136 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:54:55,136 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:55,136 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:54:55,136 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:54:55,137 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:55,137 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:55,139 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:55,139 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:55,139 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:55,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:55,139 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:55,139 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:54:55,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:55,140 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:55,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:54:55,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:55,140 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:54:55,140 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:54:55,141 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:54:55,141 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:54:55,141 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:55,141 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:54:55,141 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:54:55,141 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:54:55,141 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. 2023-04-25 02:54:55,141 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. started... 2023-04-25 02:54:55,142 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 40cf532a35b33bde962f4f28e36e887b 3/3 column families, dataSize=127.47 KB heapSize=360.38 KB 2023-04-25 02:54:55,142 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 40cf532a35b33bde962f4f28e36e887b, store=A 2023-04-25 02:54:55,142 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 02:54:55,142 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 40cf532a35b33bde962f4f28e36e887b, store=B 2023-04-25 02:54:55,142 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 02:54:55,142 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 40cf532a35b33bde962f4f28e36e887b, store=C 2023-04-25 02:54:55,142 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 02:54:55,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(9158): Flush requested on 40cf532a35b33bde962f4f28e36e887b 2023-04-25 02:54:55,215 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. as already flushing 2023-04-25 02:54:55,241 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:55,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 270 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59214 deadline: 1682391355238, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:55,291 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:55,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 357 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59158 deadline: 1682391355290, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:55,292 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:55,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 307 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59160 deadline: 1682391355290, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:55,344 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:55,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 272 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59214 deadline: 1682391355342, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:55,547 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:55,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 274 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59214 deadline: 1682391355546, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:55,553 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=936 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp/A/aab80aea7df34628ab819d26d6c743eb 2023-04-25 02:54:55,794 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:55,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 359 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59158 deadline: 1682391355792, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:55,798 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:55,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 309 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59160 deadline: 1682391355797, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:55,851 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:55,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 276 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59214 deadline: 1682391355850, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:55,967 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=936 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp/B/ca552e1c73d444469ca6eefa363f7082 2023-04-25 02:54:56,356 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:56,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 278 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59214 deadline: 1682391356355, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:56,376 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=936 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp/C/a2670746c44e4d2aa18000572b6b732c 2023-04-25 02:54:56,381 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp/A/aab80aea7df34628ab819d26d6c743eb as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A/aab80aea7df34628ab819d26d6c743eb 2023-04-25 02:54:56,385 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A/aab80aea7df34628ab819d26d6c743eb, entries=150, sequenceid=936, filesize=11.9 K 2023-04-25 02:54:56,385 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp/B/ca552e1c73d444469ca6eefa363f7082 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B/ca552e1c73d444469ca6eefa363f7082 2023-04-25 02:54:56,389 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B/ca552e1c73d444469ca6eefa363f7082, entries=150, sequenceid=936, filesize=11.9 K 2023-04-25 02:54:56,390 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp/C/a2670746c44e4d2aa18000572b6b732c as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C/a2670746c44e4d2aa18000572b6b732c 2023-04-25 02:54:56,393 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C/a2670746c44e4d2aa18000572b6b732c, entries=150, sequenceid=936, filesize=11.9 K 2023-04-25 02:54:56,394 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~360.33 KB/368976, currentSize=53.67 KB/54960 for 40cf532a35b33bde962f4f28e36e887b in 1253ms, sequenceid=936, compaction requested=true 2023-04-25 02:54:56,394 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 40cf532a35b33bde962f4f28e36e887b: 2023-04-25 02:54:56,394 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. 2023-04-25 02:54:56,394 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:54:56,394 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:54:56,394 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:56,394 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:54:56,394 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:54:56,396 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:56,396 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:56,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:56,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:56,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:56,396 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:56,396 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:54:56,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:56,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:56,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:56,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:56,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:56,397 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:54:56,397 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3bbd68e7[Count = 0] remaining members to acquire global barrier 2023-04-25 02:54:56,397 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:54:56,397 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:56,398 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:56,398 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:56,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:56,398 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:54:56,398 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:54:56,398 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:54:56,398 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:56,399 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:54:56,400 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:56,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:56,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:56,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:56,400 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:54:56,400 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:56,400 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:54:56,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:56,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:56,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:56,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:56,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:56,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:56,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:56,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:54:56,402 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:54:56,402 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:54:56,402 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:54:56,402 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:54:56,402 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:54:56,402 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:54:56,406 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:56,406 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:56,406 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:56,406 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:56,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:56,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:56,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:56,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:56,407 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:56,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:56,407 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:56,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:56,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:56,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:56,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:56,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:56,408 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:56,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:56,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:56,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:56,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:56,410 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:56,410 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:54:56,410 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:56,410 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:54:56,410 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:56,410 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:56,411 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:56,411 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:56,411 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:56,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:56,411 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:56,411 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:56,411 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:56,411 DEBUG [Thread-1512] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:54:56,411 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:54:56,411 DEBUG [Thread-1512] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:54:56,411 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:56,411 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:56,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:56,511 DEBUG [Thread-1512] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:54:56,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:54:56,513 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:54:56,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:54:56,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:54:56,514 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:54:56,514 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:56,514 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:54:56,514 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:54:56,515 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:56,515 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:56,516 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:56,516 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:56,516 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:56,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:56,516 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:56,517 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:54:56,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:56,517 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:56,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:54:56,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:56,517 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:54:56,518 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:54:56,518 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:54:56,518 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:54:56,518 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:56,518 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:54:56,518 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:54:56,518 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:54:56,518 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. 2023-04-25 02:54:56,518 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. started... 2023-04-25 02:54:56,518 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 40cf532a35b33bde962f4f28e36e887b 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-25 02:54:56,518 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 40cf532a35b33bde962f4f28e36e887b, store=A 2023-04-25 02:54:56,518 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:54:56,518 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 40cf532a35b33bde962f4f28e36e887b, store=B 2023-04-25 02:54:56,518 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:54:56,518 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 40cf532a35b33bde962f4f28e36e887b, store=C 2023-04-25 02:54:56,519 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:54:56,526 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=947 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp/A/e15f9aa450704cb3a6fa1a77b96704ae 2023-04-25 02:54:56,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(9158): Flush requested on 40cf532a35b33bde962f4f28e36e887b 2023-04-25 02:54:56,806 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. as already flushing 2023-04-25 02:54:56,838 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:56,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 319 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59160 deadline: 1682391356835, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:56,838 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682391296825, max timestamp=1682391296837Num uniques 859; 2023-04-25 02:54:56,838 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682391296825, max timestamp=1682391296837Num uniques 859; 2023-04-25 02:54:56,838 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-25 02:54:56,838 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-25 02:54:56,839 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682391296825, max timestamp=1682391296837Num uniques 859; 2023-04-25 02:54:56,839 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-25 02:54:56,884 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:56,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 382 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59158 deadline: 1682391356880, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:56,888 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:56,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 209 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59186 deadline: 1682391356883, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:56,888 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:56,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59216 deadline: 1682391356884, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:56,938 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=947 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp/B/b3f7552a80684cb58fa2cfdfa1996023 2023-04-25 02:54:56,942 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:56,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 321 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59160 deadline: 1682391356940, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:56,986 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:56,987 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 384 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59158 deadline: 1682391356985, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:56,991 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:56,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 211 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59186 deadline: 1682391356989, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:56,992 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:56,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59216 deadline: 1682391356989, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:57,146 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:57,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 323 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59160 deadline: 1682391357144, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:57,189 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:57,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 386 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59158 deadline: 1682391357188, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:57,194 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:57,194 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 213 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59186 deadline: 1682391357192, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:57,195 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:57,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59216 deadline: 1682391357193, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:57,349 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=947 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp/C/83468ec498424ed29d394f7e69152196 2023-04-25 02:54:57,353 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp/A/e15f9aa450704cb3a6fa1a77b96704ae as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A/e15f9aa450704cb3a6fa1a77b96704ae 2023-04-25 02:54:57,358 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A/e15f9aa450704cb3a6fa1a77b96704ae, entries=150, sequenceid=947, filesize=11.9 K 2023-04-25 02:54:57,358 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp/B/b3f7552a80684cb58fa2cfdfa1996023 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B/b3f7552a80684cb58fa2cfdfa1996023 2023-04-25 02:54:57,361 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:57,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 280 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59214 deadline: 1682391357357, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:57,362 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B/b3f7552a80684cb58fa2cfdfa1996023, entries=150, sequenceid=947, filesize=11.9 K 2023-04-25 02:54:57,363 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp/C/83468ec498424ed29d394f7e69152196 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C/83468ec498424ed29d394f7e69152196 2023-04-25 02:54:57,366 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C/83468ec498424ed29d394f7e69152196, entries=150, sequenceid=947, filesize=11.9 K 2023-04-25 02:54:57,367 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=134.18 KB/137400 for 40cf532a35b33bde962f4f28e36e887b in 849ms, sequenceid=947, compaction requested=true 2023-04-25 02:54:57,367 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 40cf532a35b33bde962f4f28e36e887b: 2023-04-25 02:54:57,367 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. 2023-04-25 02:54:57,367 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:54:57,367 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:54:57,367 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:57,367 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:54:57,367 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:54:57,369 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:57,369 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:57,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:57,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:57,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:57,369 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:57,369 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:54:57,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:57,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:57,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:57,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:57,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:57,371 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:54:57,371 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6d6f26bf[Count = 0] remaining members to acquire global barrier 2023-04-25 02:54:57,371 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:54:57,371 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:57,372 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:57,373 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:57,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:57,373 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:54:57,373 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:54:57,373 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:54:57,373 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:57,373 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:54:57,374 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:54:57,374 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:57,374 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:54:57,374 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:57,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:57,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:57,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:57,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:57,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:57,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:57,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:57,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:57,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:57,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:57,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:54:57,376 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:54:57,376 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:54:57,376 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:54:57,376 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:54:57,376 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:54:57,376 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:54:57,378 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:57,378 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:57,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:57,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:57,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:57,378 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:57,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:57,378 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:57,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:57,378 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:57,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:57,378 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:57,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:57,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:57,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:57,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:57,380 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:57,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:57,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:57,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:57,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:57,386 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:57,386 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:57,386 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:57,386 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:54:57,386 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:54:57,386 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:57,386 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:57,386 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:57,386 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:54:57,386 DEBUG [Thread-1512] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:54:57,386 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:57,386 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:57,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:57,386 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:57,386 DEBUG [Thread-1512] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:54:57,386 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:57,387 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:57,387 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:57,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:57,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(9158): Flush requested on 40cf532a35b33bde962f4f28e36e887b 2023-04-25 02:54:57,451 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 40cf532a35b33bde962f4f28e36e887b 3/3 column families, dataSize=140.89 KB heapSize=400.22 KB 2023-04-25 02:54:57,452 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 40cf532a35b33bde962f4f28e36e887b, store=A 2023-04-25 02:54:57,452 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 02:54:57,452 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 40cf532a35b33bde962f4f28e36e887b, store=B 2023-04-25 02:54:57,452 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 02:54:57,452 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 40cf532a35b33bde962f4f28e36e887b, store=C 2023-04-25 02:54:57,452 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 02:54:57,474 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:57,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 332 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59160 deadline: 1682391357472, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:57,487 DEBUG [Thread-1512] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:54:57,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:54:57,489 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:54:57,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:54:57,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:54:57,490 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:54:57,490 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:57,490 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:54:57,490 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:54:57,491 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:57,491 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:57,492 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:57,492 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:57,492 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:57,492 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:57,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 388 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59158 deadline: 1682391357490, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:57,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:57,492 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:57,492 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:54:57,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:57,493 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:57,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:54:57,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:57,493 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:54:57,493 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:54:57,493 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:54:57,493 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:54:57,494 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:57,494 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:54:57,494 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:54:57,494 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:54:57,494 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. 2023-04-25 02:54:57,494 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. started... 2023-04-25 02:54:57,494 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. as already flushing 2023-04-25 02:54:57,494 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. 2023-04-25 02:54:57,494 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:54:57,494 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:54:57,494 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:57,494 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:54:57,494 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:54:57,496 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:57,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:57,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:57,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:57,496 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:57,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:57,496 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:57,496 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:54:57,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:57,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:57,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:57,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:57,497 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:54:57,497 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3292561c[Count = 0] remaining members to acquire global barrier 2023-04-25 02:54:57,497 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:54:57,497 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:57,497 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:57,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 215 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59186 deadline: 1682391357495, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:57,498 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:57,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59216 deadline: 1682391357497, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:57,499 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:57,499 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:57,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:57,499 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:54:57,500 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:54:57,500 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:54:57,500 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:57,500 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:54:57,501 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:57,501 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:54:57,501 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:57,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:57,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:57,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:57,501 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:54:57,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:57,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:57,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:57,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:57,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:57,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:57,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:57,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:54:57,503 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:54:57,503 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:54:57,503 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:54:57,503 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:54:57,503 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:54:57,503 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:54:57,505 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:57,505 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:57,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:57,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:57,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:57,505 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:57,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:57,505 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:57,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:57,505 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:57,505 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:57,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:57,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:57,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:57,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:57,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:57,506 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:57,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:57,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:57,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:57,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:57,510 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:57,510 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:57,510 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:57,510 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:57,510 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:57,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:57,510 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:57,510 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:54:57,510 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:54:57,510 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:57,510 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:57,510 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:57,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:57,510 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:57,510 DEBUG [Thread-1512] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:54:57,510 DEBUG [Thread-1512] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:54:57,510 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:57,511 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:57,578 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:57,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 334 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59160 deadline: 1682391357575, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:57,611 DEBUG [Thread-1512] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:54:57,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:54:57,612 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:54:57,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:54:57,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:54:57,614 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:54:57,614 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:57,614 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:54:57,614 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:54:57,614 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:57,614 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:57,616 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:57,616 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:57,616 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:57,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:57,617 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:57,617 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:54:57,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:57,617 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:57,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:54:57,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:57,618 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:54:57,618 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:54:57,618 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:54:57,618 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:54:57,618 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:54:57,618 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:54:57,618 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:54:57,618 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:54:57,618 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. 2023-04-25 02:54:57,618 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. started... 2023-04-25 02:54:57,618 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. as already flushing 2023-04-25 02:54:57,618 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. 2023-04-25 02:54:57,618 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:54:57,619 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:54:57,619 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:57,619 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:54:57,619 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:54:57,620 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:57,620 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:57,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:57,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:57,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:57,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:57,620 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:57,620 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:54:57,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:57,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:57,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:57,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:57,622 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:54:57,622 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@109c4eb1[Count = 0] remaining members to acquire global barrier 2023-04-25 02:54:57,622 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:54:57,622 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:57,623 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:57,623 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:57,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:57,624 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:57,624 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:54:57,624 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:54:57,624 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:54:57,624 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:54:57,625 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:54:57,625 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:57,625 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:57,625 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:54:57,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:57,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:57,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:57,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:57,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:57,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:57,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:57,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:57,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:57,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:57,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:54:57,628 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:54:57,628 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:54:57,628 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:54:57,628 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:54:57,628 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:54:57,628 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:54:57,630 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:57,630 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:57,630 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:57,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:57,630 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:57,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:57,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:54:57,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:54:57,630 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:57,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:57,630 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:57,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:57,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:54:57,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:57,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:54:57,631 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:57,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:57,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:57,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:54:57,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:54:57,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:57,634 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:57,634 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:57,634 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:54:57,634 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:54:57,634 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:54:57,634 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:54:57,634 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:54:57,634 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:54:57,634 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:57,634 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:54:57,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:54:57,634 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:54:57,634 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:57,634 DEBUG [Thread-1512] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:54:57,634 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:54:57,634 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:54:57,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:54:57,634 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:54:57,634 DEBUG [Thread-1512] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:54:57,686 DEBUG [Thread-1521] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6a98265c to 127.0.0.1:62054 2023-04-25 02:54:57,686 DEBUG [Thread-1521] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-25 02:54:57,686 DEBUG [Thread-1517] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3b4658f8 to 127.0.0.1:62054 2023-04-25 02:54:57,686 DEBUG [Thread-1517] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-25 02:54:57,687 DEBUG [Thread-1515] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x66ee0bd0 to 127.0.0.1:62054 2023-04-25 02:54:57,687 DEBUG [Thread-1515] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-25 02:54:57,689 DEBUG [Thread-1519] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5d2b992d to 127.0.0.1:62054 2023-04-25 02:54:57,689 DEBUG [Thread-1519] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-25 02:54:57,689 DEBUG [Thread-1513] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x433f251d to 127.0.0.1:62054 2023-04-25 02:54:57,689 DEBUG [Thread-1513] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-25 02:54:57,734 DEBUG [Thread-1512] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:54:57,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:54:57,779 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:57,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 336 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59160 deadline: 1682391357779, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:57,860 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=984 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp/A/f5f5b02f608443d9b5dd55d747bcefb2 2023-04-25 02:54:57,997 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:57,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 390 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59158 deadline: 1682391357997, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:57,999 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:58,000 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:58,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59216 deadline: 1682391357999, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:58,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 217 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59186 deadline: 1682391357999, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:58,082 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:58,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 338 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59160 deadline: 1682391358081, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:58,270 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=984 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp/B/7834cf7a357744be94819d3ed14c6c79 2023-04-25 02:54:58,584 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:54:58,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 340 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59160 deadline: 1682391358584, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:58,680 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=984 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp/C/233e6ff660194a589437949a50fbce99 2023-04-25 02:54:58,686 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp/A/f5f5b02f608443d9b5dd55d747bcefb2 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A/f5f5b02f608443d9b5dd55d747bcefb2 2023-04-25 02:54:58,689 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A/f5f5b02f608443d9b5dd55d747bcefb2, entries=200, sequenceid=984, filesize=14.3 K 2023-04-25 02:54:58,690 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp/B/7834cf7a357744be94819d3ed14c6c79 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B/7834cf7a357744be94819d3ed14c6c79 2023-04-25 02:54:58,693 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B/7834cf7a357744be94819d3ed14c6c79, entries=150, sequenceid=984, filesize=11.9 K 2023-04-25 02:54:58,694 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp/C/233e6ff660194a589437949a50fbce99 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C/233e6ff660194a589437949a50fbce99 2023-04-25 02:54:58,697 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C/233e6ff660194a589437949a50fbce99, entries=150, sequenceid=984, filesize=11.9 K 2023-04-25 02:54:58,698 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~400.17 KB/409776, currentSize=40.25 KB/41220 for 40cf532a35b33bde962f4f28e36e887b in 1247ms, sequenceid=984, compaction requested=true 2023-04-25 02:54:58,698 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 40cf532a35b33bde962f4f28e36e887b: 2023-04-25 02:54:58,698 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 02:54:58,698 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-25 02:54:58,699 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-25 02:54:58,698 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 02:54:58,699 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 02:54:58,700 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 76510 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-25 02:54:58,700 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 74070 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-25 02:54:58,700 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HStore(1912): 40cf532a35b33bde962f4f28e36e887b/A is initiating minor compaction (all files) 2023-04-25 02:54:58,700 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HStore(1912): 40cf532a35b33bde962f4f28e36e887b/B is initiating minor compaction (all files) 2023-04-25 02:54:58,700 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 40cf532a35b33bde962f4f28e36e887b/A in TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. 2023-04-25 02:54:58,700 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 40cf532a35b33bde962f4f28e36e887b/B in TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. 2023-04-25 02:54:58,700 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A/6aa5474d0e4b485bb50d172e4a656210, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A/5dd3a2cc21f240f0b9edd04b30b6f144, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A/febc3a0bd39344d889abb5673b6660fb, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A/aab80aea7df34628ab819d26d6c743eb, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A/e15f9aa450704cb3a6fa1a77b96704ae, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A/f5f5b02f608443d9b5dd55d747bcefb2] into tmpdir=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp, totalSize=74.7 K 2023-04-25 02:54:58,701 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B/5041509541624492accd344faa62e0d3, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B/8d0bbdd1546346ecba47c226ac7e9b7d, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B/9a04fd7bb7224b39b8d90a74d0a53681, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B/ca552e1c73d444469ca6eefa363f7082, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B/b3f7552a80684cb58fa2cfdfa1996023, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B/7834cf7a357744be94819d3ed14c6c79] into tmpdir=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp, totalSize=72.3 K 2023-04-25 02:54:58,701 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting 6aa5474d0e4b485bb50d172e4a656210, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=850, earliestPutTs=1682391292827 2023-04-25 02:54:58,701 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting 5041509541624492accd344faa62e0d3, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=850, earliestPutTs=1682391292827 2023-04-25 02:54:58,701 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting 5dd3a2cc21f240f0b9edd04b30b6f144, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=887, earliestPutTs=1682391293499 2023-04-25 02:54:58,701 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting 8d0bbdd1546346ecba47c226ac7e9b7d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=887, earliestPutTs=1682391293499 2023-04-25 02:54:58,701 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting febc3a0bd39344d889abb5673b6660fb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=901, earliestPutTs=1682391294089 2023-04-25 02:54:58,702 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting 9a04fd7bb7224b39b8d90a74d0a53681, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=901, earliestPutTs=1682391294089 2023-04-25 02:54:58,702 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting aab80aea7df34628ab819d26d6c743eb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=936, earliestPutTs=1682391294672 2023-04-25 02:54:58,702 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting ca552e1c73d444469ca6eefa363f7082, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=936, earliestPutTs=1682391294672 2023-04-25 02:54:58,702 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting e15f9aa450704cb3a6fa1a77b96704ae, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=947, earliestPutTs=1682391295228 2023-04-25 02:54:58,702 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting b3f7552a80684cb58fa2cfdfa1996023, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=947, earliestPutTs=1682391295228 2023-04-25 02:54:58,702 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting f5f5b02f608443d9b5dd55d747bcefb2, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=984, earliestPutTs=1682391296876 2023-04-25 02:54:58,702 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting 7834cf7a357744be94819d3ed14c6c79, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=984, earliestPutTs=1682391296882 2023-04-25 02:54:58,713 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] throttle.PressureAwareThroughputController(145): 40cf532a35b33bde962f4f28e36e887b#A#compaction#285 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 53.85 MB/second 2023-04-25 02:54:58,713 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] throttle.PressureAwareThroughputController(145): 40cf532a35b33bde962f4f28e36e887b#B#compaction#286 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-25 02:54:59,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(9158): Flush requested on 40cf532a35b33bde962f4f28e36e887b 2023-04-25 02:54:59,002 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 40cf532a35b33bde962f4f28e36e887b 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-25 02:54:59,002 DEBUG [Thread-1502] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3157d4df to 127.0.0.1:62054 2023-04-25 02:54:59,002 DEBUG [Thread-1502] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-25 02:54:59,002 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 40cf532a35b33bde962f4f28e36e887b, store=A 2023-04-25 02:54:59,002 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:54:59,002 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 40cf532a35b33bde962f4f28e36e887b, store=B 2023-04-25 02:54:59,002 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:54:59,002 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 40cf532a35b33bde962f4f28e36e887b, store=C 2023-04-25 02:54:59,002 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:54:59,004 DEBUG [Thread-1510] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x087b3095 to 127.0.0.1:62054 2023-04-25 02:54:59,004 DEBUG [Thread-1510] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-25 02:54:59,007 DEBUG [Thread-1504] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3cf5bdb7 to 127.0.0.1:62054 2023-04-25 02:54:59,008 DEBUG [Thread-1504] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-25 02:54:59,120 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp/A/95e273d3596b4b6e988ccbcfa43f5aaa as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A/95e273d3596b4b6e988ccbcfa43f5aaa 2023-04-25 02:54:59,124 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 40cf532a35b33bde962f4f28e36e887b/A of 40cf532a35b33bde962f4f28e36e887b into 95e273d3596b4b6e988ccbcfa43f5aaa(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 02:54:59,124 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 40cf532a35b33bde962f4f28e36e887b: 2023-04-25 02:54:59,124 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b., storeName=40cf532a35b33bde962f4f28e36e887b/A, priority=10, startTime=1682391298698; duration=0sec 2023-04-25 02:54:59,124 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 02:54:59,125 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-25 02:54:59,126 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 74070 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-25 02:54:59,126 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HStore(1912): 40cf532a35b33bde962f4f28e36e887b/C is initiating minor compaction (all files) 2023-04-25 02:54:59,126 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 40cf532a35b33bde962f4f28e36e887b/C in TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. 2023-04-25 02:54:59,126 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C/f78bd380cc2c4215b6527e71f3259150, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C/4a82ea8131454324850d359e769b09f5, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C/e94e9f097d5448a59f5dfa2babd43a9d, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C/a2670746c44e4d2aa18000572b6b732c, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C/83468ec498424ed29d394f7e69152196, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C/233e6ff660194a589437949a50fbce99] into tmpdir=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp, totalSize=72.3 K 2023-04-25 02:54:59,127 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting f78bd380cc2c4215b6527e71f3259150, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=850, earliestPutTs=1682391292827 2023-04-25 02:54:59,127 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting 4a82ea8131454324850d359e769b09f5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=887, earliestPutTs=1682391293499 2023-04-25 02:54:59,127 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting e94e9f097d5448a59f5dfa2babd43a9d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=901, earliestPutTs=1682391294089 2023-04-25 02:54:59,128 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting a2670746c44e4d2aa18000572b6b732c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=936, earliestPutTs=1682391294672 2023-04-25 02:54:59,128 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting 83468ec498424ed29d394f7e69152196, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=947, earliestPutTs=1682391295228 2023-04-25 02:54:59,128 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting 233e6ff660194a589437949a50fbce99, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=984, earliestPutTs=1682391296882 2023-04-25 02:54:59,128 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp/B/8d7d6a55b5214cb9acd319835380e98b as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B/8d7d6a55b5214cb9acd319835380e98b 2023-04-25 02:54:59,133 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 40cf532a35b33bde962f4f28e36e887b/B of 40cf532a35b33bde962f4f28e36e887b into 8d7d6a55b5214cb9acd319835380e98b(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 02:54:59,133 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 40cf532a35b33bde962f4f28e36e887b: 2023-04-25 02:54:59,133 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b., storeName=40cf532a35b33bde962f4f28e36e887b/B, priority=10, startTime=1682391298698; duration=0sec 2023-04-25 02:54:59,133 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 02:54:59,137 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] throttle.PressureAwareThroughputController(145): 40cf532a35b33bde962f4f28e36e887b#C#compaction#288 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-25 02:54:59,372 DEBUG [Thread-1508] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2ce5573e to 127.0.0.1:62054 2023-04-25 02:54:59,372 DEBUG [Thread-1508] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-25 02:54:59,409 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=994 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp/A/f3a397e14602453e919af8abccc4287c 2023-04-25 02:54:59,545 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp/C/c33192b45bc4488b828676928d7d4cdb as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C/c33192b45bc4488b828676928d7d4cdb 2023-04-25 02:54:59,549 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 40cf532a35b33bde962f4f28e36e887b/C of 40cf532a35b33bde962f4f28e36e887b into c33192b45bc4488b828676928d7d4cdb(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 02:54:59,549 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 40cf532a35b33bde962f4f28e36e887b: 2023-04-25 02:54:59,549 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b., storeName=40cf532a35b33bde962f4f28e36e887b/C, priority=10, startTime=1682391298699; duration=0sec 2023-04-25 02:54:59,549 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 02:54:59,592 DEBUG [Thread-1506] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6919b2a0 to 127.0.0.1:62054 2023-04-25 02:54:59,592 DEBUG [Thread-1506] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-25 02:54:59,592 INFO [Listener at localhost/41915] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-25 02:54:59,593 INFO [Listener at localhost/41915] hbase.AcidGuaranteesTestTool(394): wrote 107 2023-04-25 02:54:59,593 INFO [Listener at localhost/41915] hbase.AcidGuaranteesTestTool(394): wrote 126 2023-04-25 02:54:59,593 INFO [Listener at localhost/41915] hbase.AcidGuaranteesTestTool(394): wrote 221 2023-04-25 02:54:59,593 INFO [Listener at localhost/41915] hbase.AcidGuaranteesTestTool(394): wrote 155 2023-04-25 02:54:59,593 INFO [Listener at localhost/41915] hbase.AcidGuaranteesTestTool(394): wrote 283 2023-04-25 02:54:59,593 INFO [Listener at localhost/41915] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-25 02:54:59,593 INFO [Listener at localhost/41915] hbase.AcidGuaranteesTestTool(398): read 4858 2023-04-25 02:54:59,593 INFO [Listener at localhost/41915] hbase.AcidGuaranteesTestTool(398): read 4843 2023-04-25 02:54:59,593 INFO [Listener at localhost/41915] hbase.AcidGuaranteesTestTool(398): read 4924 2023-04-25 02:54:59,593 INFO [Listener at localhost/41915] hbase.AcidGuaranteesTestTool(398): read 4888 2023-04-25 02:54:59,593 INFO [Listener at localhost/41915] hbase.AcidGuaranteesTestTool(398): read 4862 2023-04-25 02:54:59,593 INFO [Listener at localhost/41915] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-25 02:54:59,593 INFO [Listener at localhost/41915] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-25 02:54:59,593 DEBUG [Listener at localhost/41915] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5c124820 to 127.0.0.1:62054 2023-04-25 02:54:59,593 DEBUG [Listener at localhost/41915] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-25 02:54:59,594 INFO [Listener at localhost/41915] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-25 02:54:59,594 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-04-25 02:54:59,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure2.ProcedureExecutor(1029): Stored pid=31, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-25 02:54:59,597 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-25 02:54:59,597 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682391299597"}]},"ts":"1682391299597"} 2023-04-25 02:54:59,598 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-25 02:54:59,600 INFO [PEWorker-3] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-25 02:54:59,600 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=32, ppid=31, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=40cf532a35b33bde962f4f28e36e887b, UNASSIGN}] 2023-04-25 02:54:59,602 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=32, ppid=31, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=40cf532a35b33bde962f4f28e36e887b, UNASSIGN 2023-04-25 02:54:59,602 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=32 updating hbase:meta row=40cf532a35b33bde962f4f28e36e887b, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:54:59,602 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682391299602"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682391299602"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682391299602"}]},"ts":"1682391299602"} 2023-04-25 02:54:59,603 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=33, ppid=32, state=RUNNABLE; CloseRegionProcedure 40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023}] 2023-04-25 02:54:59,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-25 02:54:59,755 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 40cf532a35b33bde962f4f28e36e887b 2023-04-25 02:54:59,755 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 40cf532a35b33bde962f4f28e36e887b, disabling compactions & flushes 2023-04-25 02:54:59,755 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1858): waiting for 0 compactions & cache flush to complete for region TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. 2023-04-25 02:54:59,818 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=994 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp/B/8ddef905bf0d47498cee33327836b443 2023-04-25 02:54:59,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-25 02:55:00,199 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-25 02:55:00,227 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=994 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp/C/ab571f1cfe5a4e788e3e6b25a951b61a 2023-04-25 02:55:00,231 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp/A/f3a397e14602453e919af8abccc4287c as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A/f3a397e14602453e919af8abccc4287c 2023-04-25 02:55:00,234 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A/f3a397e14602453e919af8abccc4287c, entries=150, sequenceid=994, filesize=11.9 K 2023-04-25 02:55:00,234 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp/B/8ddef905bf0d47498cee33327836b443 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B/8ddef905bf0d47498cee33327836b443 2023-04-25 02:55:00,238 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B/8ddef905bf0d47498cee33327836b443, entries=150, sequenceid=994, filesize=11.9 K 2023-04-25 02:55:00,238 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp/C/ab571f1cfe5a4e788e3e6b25a951b61a as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C/ab571f1cfe5a4e788e3e6b25a951b61a 2023-04-25 02:55:00,242 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C/ab571f1cfe5a4e788e3e6b25a951b61a, entries=150, sequenceid=994, filesize=11.9 K 2023-04-25 02:55:00,242 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=26.84 KB/27480 for 40cf532a35b33bde962f4f28e36e887b in 1240ms, sequenceid=994, compaction requested=false 2023-04-25 02:55:00,242 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 40cf532a35b33bde962f4f28e36e887b: 2023-04-25 02:55:00,243 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. 2023-04-25 02:55:00,243 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. 2023-04-25 02:55:00,243 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. after waiting 0 ms 2023-04-25 02:55:00,243 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. 2023-04-25 02:55:00,243 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 40cf532a35b33bde962f4f28e36e887b 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-25 02:55:00,243 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 40cf532a35b33bde962f4f28e36e887b, store=A 2023-04-25 02:55:00,243 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:55:00,243 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 40cf532a35b33bde962f4f28e36e887b, store=B 2023-04-25 02:55:00,243 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:55:00,243 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 40cf532a35b33bde962f4f28e36e887b, store=C 2023-04-25 02:55:00,243 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:55:00,649 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=1004 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp/A/d6521da165cc47ff8075e84093423798 2023-04-25 02:55:00,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-25 02:55:01,059 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=1004 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp/B/bb311720ce02415aa852c1671edc2c00 2023-04-25 02:55:01,468 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=1004 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp/C/e694a6fddbce4541b859647259741f4f 2023-04-25 02:55:01,471 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp/A/d6521da165cc47ff8075e84093423798 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A/d6521da165cc47ff8075e84093423798 2023-04-25 02:55:01,474 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A/d6521da165cc47ff8075e84093423798, entries=100, sequenceid=1004, filesize=9.5 K 2023-04-25 02:55:01,475 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp/B/bb311720ce02415aa852c1671edc2c00 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B/bb311720ce02415aa852c1671edc2c00 2023-04-25 02:55:01,478 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B/bb311720ce02415aa852c1671edc2c00, entries=100, sequenceid=1004, filesize=9.5 K 2023-04-25 02:55:01,478 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/.tmp/C/e694a6fddbce4541b859647259741f4f as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C/e694a6fddbce4541b859647259741f4f 2023-04-25 02:55:01,481 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C/e694a6fddbce4541b859647259741f4f, entries=100, sequenceid=1004, filesize=9.5 K 2023-04-25 02:55:01,482 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=0 B/0 for 40cf532a35b33bde962f4f28e36e887b in 1239ms, sequenceid=1004, compaction requested=true 2023-04-25 02:55:01,503 DEBUG [StoreCloser-TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A/92c9b725031443e5b38491daade551ee, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A/eeace9228e5749bca7c0ad6499f5263e, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A/56fdb1660bae4bcd935540ba6f54b811, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A/0104577a6bdf452cae48db5ff23270f6, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A/3bec4c32bf214871996921c22931d36b, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A/a5d3cbd452244da5a1c55147119f7e0a, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A/a9bb2db4c08e4bccbbd7ed61c73381ea, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A/41bcf169dead4de3b85c0b95ee2d8423, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A/a0a64b540ec94a6b8d5b262e32b70b43, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A/c4734762cf6644aeab94a7ebd511bb1a, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A/ff82e765070b41c2844f1d44d154a8bb, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A/3204165bcff647c1a5dc2becf3becf8d, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A/b004de39cde34321ad418bb4c33804e6, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A/cbe8e6cba8e04939a8da94367ec0e635, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A/f80956a04fd44e29953e6f8db595c477, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A/818b8e53d66a4bc1b70ccda74e5a67a9, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A/9dfeafb5844843d2a4f4d1193640fc58, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A/15decf60ca8c4eebb169da7532e5b1b8, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A/bfad6c23c6114861a1006cf7314ee700, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A/e5225d64e6aa4b3f96fa5d19454ef470, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A/2399b71f9c3344ad9483c17ec0757716, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A/2b5a29dae9074175b6caedaebc256abc, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A/6d781aadadbf44b98ee0c11862c78b54, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A/c46e0fe7a3be483e838d90c690bd969c, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A/2e2e8c94bc304019a39d9caa97d8a251, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A/fe041106d1da491890c6251e54d49bf7, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A/9216d1ac67964930bc892bf6cdd5da91, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A/6aa5474d0e4b485bb50d172e4a656210, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A/fa3a7b9844c34827a46099a848c2fb05, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A/5dd3a2cc21f240f0b9edd04b30b6f144, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A/febc3a0bd39344d889abb5673b6660fb, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A/aab80aea7df34628ab819d26d6c743eb, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A/e15f9aa450704cb3a6fa1a77b96704ae, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A/f5f5b02f608443d9b5dd55d747bcefb2] to archive 2023-04-25 02:55:01,504 DEBUG [StoreCloser-TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-25 02:55:01,505 DEBUG [StoreCloser-TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A/92c9b725031443e5b38491daade551ee to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A/92c9b725031443e5b38491daade551ee 2023-04-25 02:55:01,506 DEBUG [StoreCloser-TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A/eeace9228e5749bca7c0ad6499f5263e to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A/eeace9228e5749bca7c0ad6499f5263e 2023-04-25 02:55:01,507 DEBUG [StoreCloser-TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A/56fdb1660bae4bcd935540ba6f54b811 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A/56fdb1660bae4bcd935540ba6f54b811 2023-04-25 02:55:01,508 DEBUG [StoreCloser-TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A/0104577a6bdf452cae48db5ff23270f6 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A/0104577a6bdf452cae48db5ff23270f6 2023-04-25 02:55:01,509 DEBUG [StoreCloser-TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A/3bec4c32bf214871996921c22931d36b to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A/3bec4c32bf214871996921c22931d36b 2023-04-25 02:55:01,510 DEBUG [StoreCloser-TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A/a5d3cbd452244da5a1c55147119f7e0a to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A/a5d3cbd452244da5a1c55147119f7e0a 2023-04-25 02:55:01,511 DEBUG [StoreCloser-TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A/a9bb2db4c08e4bccbbd7ed61c73381ea to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A/a9bb2db4c08e4bccbbd7ed61c73381ea 2023-04-25 02:55:01,512 DEBUG [StoreCloser-TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A/41bcf169dead4de3b85c0b95ee2d8423 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A/41bcf169dead4de3b85c0b95ee2d8423 2023-04-25 02:55:01,513 DEBUG [StoreCloser-TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A/a0a64b540ec94a6b8d5b262e32b70b43 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A/a0a64b540ec94a6b8d5b262e32b70b43 2023-04-25 02:55:01,514 DEBUG [StoreCloser-TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A/c4734762cf6644aeab94a7ebd511bb1a to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A/c4734762cf6644aeab94a7ebd511bb1a 2023-04-25 02:55:01,515 DEBUG [StoreCloser-TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A/ff82e765070b41c2844f1d44d154a8bb to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A/ff82e765070b41c2844f1d44d154a8bb 2023-04-25 02:55:01,516 DEBUG [StoreCloser-TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A/3204165bcff647c1a5dc2becf3becf8d to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A/3204165bcff647c1a5dc2becf3becf8d 2023-04-25 02:55:01,517 DEBUG [StoreCloser-TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A/b004de39cde34321ad418bb4c33804e6 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A/b004de39cde34321ad418bb4c33804e6 2023-04-25 02:55:01,518 DEBUG [StoreCloser-TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A/cbe8e6cba8e04939a8da94367ec0e635 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A/cbe8e6cba8e04939a8da94367ec0e635 2023-04-25 02:55:01,519 DEBUG [StoreCloser-TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A/f80956a04fd44e29953e6f8db595c477 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A/f80956a04fd44e29953e6f8db595c477 2023-04-25 02:55:01,519 DEBUG [StoreCloser-TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A/818b8e53d66a4bc1b70ccda74e5a67a9 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A/818b8e53d66a4bc1b70ccda74e5a67a9 2023-04-25 02:55:01,520 DEBUG [StoreCloser-TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A/9dfeafb5844843d2a4f4d1193640fc58 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A/9dfeafb5844843d2a4f4d1193640fc58 2023-04-25 02:55:01,521 DEBUG [StoreCloser-TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A/15decf60ca8c4eebb169da7532e5b1b8 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A/15decf60ca8c4eebb169da7532e5b1b8 2023-04-25 02:55:01,522 DEBUG [StoreCloser-TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A/bfad6c23c6114861a1006cf7314ee700 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A/bfad6c23c6114861a1006cf7314ee700 2023-04-25 02:55:01,523 DEBUG [StoreCloser-TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A/e5225d64e6aa4b3f96fa5d19454ef470 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A/e5225d64e6aa4b3f96fa5d19454ef470 2023-04-25 02:55:01,524 DEBUG [StoreCloser-TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A/2399b71f9c3344ad9483c17ec0757716 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A/2399b71f9c3344ad9483c17ec0757716 2023-04-25 02:55:01,525 DEBUG [StoreCloser-TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A/2b5a29dae9074175b6caedaebc256abc to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A/2b5a29dae9074175b6caedaebc256abc 2023-04-25 02:55:01,525 DEBUG [StoreCloser-TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A/6d781aadadbf44b98ee0c11862c78b54 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A/6d781aadadbf44b98ee0c11862c78b54 2023-04-25 02:55:01,526 DEBUG [StoreCloser-TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A/c46e0fe7a3be483e838d90c690bd969c to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A/c46e0fe7a3be483e838d90c690bd969c 2023-04-25 02:55:01,527 DEBUG [StoreCloser-TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A/2e2e8c94bc304019a39d9caa97d8a251 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A/2e2e8c94bc304019a39d9caa97d8a251 2023-04-25 02:55:01,527 DEBUG [StoreCloser-TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A/fe041106d1da491890c6251e54d49bf7 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A/fe041106d1da491890c6251e54d49bf7 2023-04-25 02:55:01,528 DEBUG [StoreCloser-TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A/9216d1ac67964930bc892bf6cdd5da91 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A/9216d1ac67964930bc892bf6cdd5da91 2023-04-25 02:55:01,529 DEBUG [StoreCloser-TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A/6aa5474d0e4b485bb50d172e4a656210 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A/6aa5474d0e4b485bb50d172e4a656210 2023-04-25 02:55:01,530 DEBUG [StoreCloser-TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A/fa3a7b9844c34827a46099a848c2fb05 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A/fa3a7b9844c34827a46099a848c2fb05 2023-04-25 02:55:01,530 DEBUG [StoreCloser-TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A/5dd3a2cc21f240f0b9edd04b30b6f144 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A/5dd3a2cc21f240f0b9edd04b30b6f144 2023-04-25 02:55:01,531 DEBUG [StoreCloser-TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A/febc3a0bd39344d889abb5673b6660fb to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A/febc3a0bd39344d889abb5673b6660fb 2023-04-25 02:55:01,532 DEBUG [StoreCloser-TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A/aab80aea7df34628ab819d26d6c743eb to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A/aab80aea7df34628ab819d26d6c743eb 2023-04-25 02:55:01,532 DEBUG [StoreCloser-TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A/e15f9aa450704cb3a6fa1a77b96704ae to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A/e15f9aa450704cb3a6fa1a77b96704ae 2023-04-25 02:55:01,533 DEBUG [StoreCloser-TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A/f5f5b02f608443d9b5dd55d747bcefb2 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A/f5f5b02f608443d9b5dd55d747bcefb2 2023-04-25 02:55:01,536 DEBUG [StoreCloser-TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B/370a415aaab24a6da9e885fe53ad6176, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B/f11aeab20c8b4454bc900c2fa1d8ad0c, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B/b373fbd5f50c43c1834fd6842d856498, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B/f5926eca392e48c6956392422ed54275, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B/cd0f47406b3e4b259e501639ad4f11c9, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B/9e9aa9ffa66f4e58b85b838149872849, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B/b5855a635f154736af1bb491f09ecd2b, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B/395cb4f2860540d08f38f5626a6c12c5, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B/10af95acf8654acaaf02994c65165195, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B/9d666fbce5414c1eb8a974973d4ee483, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B/3b3b650ac318479eae62b99d045f88aa, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B/76482a48794d46a3ab6ff78f6a7ab9da, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B/47cbd8623b5b4bf6a7b97e67c55451fd, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B/8975a22ef3e24d10bc69c68356164b57, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B/6c05444a679b4b46a4b38a7569e2cf74, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B/e45a857578e9461382e39d5313efd5df, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B/dba1c17f78134c6fb3ad8ccf325ee940, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B/bebfc9606c914037beea7f4da2d95826, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B/08dca444144045e6be9e4676c13fdf9e, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B/74e8ea961ae2425c8c7f9403bf29ad90, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B/3c879b9204e144a1afb40fee4449ae89, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B/76ce613289ae4db386e81e6e16d7c4ce, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B/ae0e1db7e0874f4cae02d06057d671dd, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B/53dfa555a4a74f9f839d959aae43382a, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B/6ab5c5473c38478397548c94668ae5ea, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B/a92547c229664cbf8f7ccd238769a2c6, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B/c28fe9c09f8d40ec8e2ad2d0ee6e1b23, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B/5041509541624492accd344faa62e0d3, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B/5f9e6a0a308344a59b01681fd7734911, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B/8d0bbdd1546346ecba47c226ac7e9b7d, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B/9a04fd7bb7224b39b8d90a74d0a53681, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B/ca552e1c73d444469ca6eefa363f7082, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B/b3f7552a80684cb58fa2cfdfa1996023, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B/7834cf7a357744be94819d3ed14c6c79] to archive 2023-04-25 02:55:01,537 DEBUG [StoreCloser-TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-25 02:55:01,538 DEBUG [StoreCloser-TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B/370a415aaab24a6da9e885fe53ad6176 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B/370a415aaab24a6da9e885fe53ad6176 2023-04-25 02:55:01,539 DEBUG [StoreCloser-TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B/f11aeab20c8b4454bc900c2fa1d8ad0c to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B/f11aeab20c8b4454bc900c2fa1d8ad0c 2023-04-25 02:55:01,539 DEBUG [StoreCloser-TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B/b373fbd5f50c43c1834fd6842d856498 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B/b373fbd5f50c43c1834fd6842d856498 2023-04-25 02:55:01,540 DEBUG [StoreCloser-TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B/f5926eca392e48c6956392422ed54275 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B/f5926eca392e48c6956392422ed54275 2023-04-25 02:55:01,541 DEBUG [StoreCloser-TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B/cd0f47406b3e4b259e501639ad4f11c9 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B/cd0f47406b3e4b259e501639ad4f11c9 2023-04-25 02:55:01,542 DEBUG [StoreCloser-TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B/9e9aa9ffa66f4e58b85b838149872849 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B/9e9aa9ffa66f4e58b85b838149872849 2023-04-25 02:55:01,543 DEBUG [StoreCloser-TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B/b5855a635f154736af1bb491f09ecd2b to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B/b5855a635f154736af1bb491f09ecd2b 2023-04-25 02:55:01,544 DEBUG [StoreCloser-TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B/395cb4f2860540d08f38f5626a6c12c5 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B/395cb4f2860540d08f38f5626a6c12c5 2023-04-25 02:55:01,545 DEBUG [StoreCloser-TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B/10af95acf8654acaaf02994c65165195 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B/10af95acf8654acaaf02994c65165195 2023-04-25 02:55:01,545 DEBUG [StoreCloser-TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B/9d666fbce5414c1eb8a974973d4ee483 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B/9d666fbce5414c1eb8a974973d4ee483 2023-04-25 02:55:01,546 DEBUG [StoreCloser-TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B/3b3b650ac318479eae62b99d045f88aa to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B/3b3b650ac318479eae62b99d045f88aa 2023-04-25 02:55:01,547 DEBUG [StoreCloser-TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B/76482a48794d46a3ab6ff78f6a7ab9da to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B/76482a48794d46a3ab6ff78f6a7ab9da 2023-04-25 02:55:01,548 DEBUG [StoreCloser-TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B/47cbd8623b5b4bf6a7b97e67c55451fd to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B/47cbd8623b5b4bf6a7b97e67c55451fd 2023-04-25 02:55:01,548 DEBUG [StoreCloser-TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B/8975a22ef3e24d10bc69c68356164b57 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B/8975a22ef3e24d10bc69c68356164b57 2023-04-25 02:55:01,549 DEBUG [StoreCloser-TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B/6c05444a679b4b46a4b38a7569e2cf74 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B/6c05444a679b4b46a4b38a7569e2cf74 2023-04-25 02:55:01,550 DEBUG [StoreCloser-TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B/e45a857578e9461382e39d5313efd5df to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B/e45a857578e9461382e39d5313efd5df 2023-04-25 02:55:01,551 DEBUG [StoreCloser-TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B/dba1c17f78134c6fb3ad8ccf325ee940 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B/dba1c17f78134c6fb3ad8ccf325ee940 2023-04-25 02:55:01,552 DEBUG [StoreCloser-TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B/bebfc9606c914037beea7f4da2d95826 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B/bebfc9606c914037beea7f4da2d95826 2023-04-25 02:55:01,553 DEBUG [StoreCloser-TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B/08dca444144045e6be9e4676c13fdf9e to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B/08dca444144045e6be9e4676c13fdf9e 2023-04-25 02:55:01,554 DEBUG [StoreCloser-TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B/74e8ea961ae2425c8c7f9403bf29ad90 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B/74e8ea961ae2425c8c7f9403bf29ad90 2023-04-25 02:55:01,554 DEBUG [StoreCloser-TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B/3c879b9204e144a1afb40fee4449ae89 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B/3c879b9204e144a1afb40fee4449ae89 2023-04-25 02:55:01,555 DEBUG [StoreCloser-TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B/76ce613289ae4db386e81e6e16d7c4ce to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B/76ce613289ae4db386e81e6e16d7c4ce 2023-04-25 02:55:01,556 DEBUG [StoreCloser-TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B/ae0e1db7e0874f4cae02d06057d671dd to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B/ae0e1db7e0874f4cae02d06057d671dd 2023-04-25 02:55:01,557 DEBUG [StoreCloser-TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B/53dfa555a4a74f9f839d959aae43382a to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B/53dfa555a4a74f9f839d959aae43382a 2023-04-25 02:55:01,558 DEBUG [StoreCloser-TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B/6ab5c5473c38478397548c94668ae5ea to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B/6ab5c5473c38478397548c94668ae5ea 2023-04-25 02:55:01,559 DEBUG [StoreCloser-TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B/a92547c229664cbf8f7ccd238769a2c6 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B/a92547c229664cbf8f7ccd238769a2c6 2023-04-25 02:55:01,559 DEBUG [StoreCloser-TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B/c28fe9c09f8d40ec8e2ad2d0ee6e1b23 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B/c28fe9c09f8d40ec8e2ad2d0ee6e1b23 2023-04-25 02:55:01,560 DEBUG [StoreCloser-TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B/5041509541624492accd344faa62e0d3 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B/5041509541624492accd344faa62e0d3 2023-04-25 02:55:01,561 DEBUG [StoreCloser-TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B/5f9e6a0a308344a59b01681fd7734911 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B/5f9e6a0a308344a59b01681fd7734911 2023-04-25 02:55:01,562 DEBUG [StoreCloser-TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B/8d0bbdd1546346ecba47c226ac7e9b7d to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B/8d0bbdd1546346ecba47c226ac7e9b7d 2023-04-25 02:55:01,563 DEBUG [StoreCloser-TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B/9a04fd7bb7224b39b8d90a74d0a53681 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B/9a04fd7bb7224b39b8d90a74d0a53681 2023-04-25 02:55:01,564 DEBUG [StoreCloser-TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B/ca552e1c73d444469ca6eefa363f7082 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B/ca552e1c73d444469ca6eefa363f7082 2023-04-25 02:55:01,565 DEBUG [StoreCloser-TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B/b3f7552a80684cb58fa2cfdfa1996023 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B/b3f7552a80684cb58fa2cfdfa1996023 2023-04-25 02:55:01,566 DEBUG [StoreCloser-TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B/7834cf7a357744be94819d3ed14c6c79 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B/7834cf7a357744be94819d3ed14c6c79 2023-04-25 02:55:01,568 DEBUG [StoreCloser-TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C/017f6cf1430b4dc5a4b4e605ead65115, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C/621545d991884cd6bda0d135d0fbfff9, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C/47f6457404874cc5943e7bcb2e8c22c1, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C/3f6bb3b42dc8411a848d42d32daecea6, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C/ab039aa53efc4464a6044bb43d22e721, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C/990b63ee975c4703b2e5ae29b7fe5c53, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C/caf18a24d2864a2c9252a73580f5ee3e, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C/f5175a01513c4b729bc1ded565a8dc40, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C/accc96103a574b0d898f2420efedbf6c, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C/45ec236adc5e40049ce316032a7a3e21, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C/851700bd208e4350b1eb976dc306e1a5, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C/20d9ec3f2ab341608e7313227e1ffbf8, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C/7ee5c141bc454854a59b483fc5a6028d, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C/08a16b8aa41a4f5da673c932b5a8b41c, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C/aa98107974264ee8b9ef7be771d8e5e5, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C/c8591ba6d2734211b1962f3142544a13, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C/acc14600f8304afd87bda8566ab499eb, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C/17c8516514d84ca69b3424137388e781, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C/cd8f31c2e6a94246a6a3dc333e52e680, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C/5df0c28533b544ecaac582e943da5814, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C/0dee40da444048a098e3730c0e1f7e28, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C/7c2a1f49b23a4fe08ab8c2567609d5aa, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C/5c76b8d71c534b07946f1548b5df5a4f, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C/a53a6bec412a44d2aaeffb999e63b6a3, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C/ca9b6e0a629f4db186e5b8a47293b8fe, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C/b06f0f1632084ffda1167091ff512f6e, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C/2388ec51e2ea4ddd804098b60f33c29c, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C/f78bd380cc2c4215b6527e71f3259150, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C/37f186c853bb4d1bbc4c5996b10e31c2, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C/4a82ea8131454324850d359e769b09f5, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C/e94e9f097d5448a59f5dfa2babd43a9d, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C/a2670746c44e4d2aa18000572b6b732c, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C/83468ec498424ed29d394f7e69152196, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C/233e6ff660194a589437949a50fbce99] to archive 2023-04-25 02:55:01,569 DEBUG [StoreCloser-TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-25 02:55:01,570 DEBUG [StoreCloser-TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C/017f6cf1430b4dc5a4b4e605ead65115 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C/017f6cf1430b4dc5a4b4e605ead65115 2023-04-25 02:55:01,571 DEBUG [StoreCloser-TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C/621545d991884cd6bda0d135d0fbfff9 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C/621545d991884cd6bda0d135d0fbfff9 2023-04-25 02:55:01,572 DEBUG [StoreCloser-TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C/47f6457404874cc5943e7bcb2e8c22c1 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C/47f6457404874cc5943e7bcb2e8c22c1 2023-04-25 02:55:01,573 DEBUG [StoreCloser-TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C/3f6bb3b42dc8411a848d42d32daecea6 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C/3f6bb3b42dc8411a848d42d32daecea6 2023-04-25 02:55:01,573 DEBUG [StoreCloser-TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C/ab039aa53efc4464a6044bb43d22e721 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C/ab039aa53efc4464a6044bb43d22e721 2023-04-25 02:55:01,574 DEBUG [StoreCloser-TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C/990b63ee975c4703b2e5ae29b7fe5c53 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C/990b63ee975c4703b2e5ae29b7fe5c53 2023-04-25 02:55:01,575 DEBUG [StoreCloser-TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C/caf18a24d2864a2c9252a73580f5ee3e to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C/caf18a24d2864a2c9252a73580f5ee3e 2023-04-25 02:55:01,576 DEBUG [StoreCloser-TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C/f5175a01513c4b729bc1ded565a8dc40 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C/f5175a01513c4b729bc1ded565a8dc40 2023-04-25 02:55:01,576 DEBUG [StoreCloser-TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C/accc96103a574b0d898f2420efedbf6c to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C/accc96103a574b0d898f2420efedbf6c 2023-04-25 02:55:01,577 DEBUG [StoreCloser-TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C/45ec236adc5e40049ce316032a7a3e21 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C/45ec236adc5e40049ce316032a7a3e21 2023-04-25 02:55:01,578 DEBUG [StoreCloser-TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C/851700bd208e4350b1eb976dc306e1a5 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C/851700bd208e4350b1eb976dc306e1a5 2023-04-25 02:55:01,579 DEBUG [StoreCloser-TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C/20d9ec3f2ab341608e7313227e1ffbf8 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C/20d9ec3f2ab341608e7313227e1ffbf8 2023-04-25 02:55:01,579 DEBUG [StoreCloser-TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C/7ee5c141bc454854a59b483fc5a6028d to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C/7ee5c141bc454854a59b483fc5a6028d 2023-04-25 02:55:01,580 DEBUG [StoreCloser-TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C/08a16b8aa41a4f5da673c932b5a8b41c to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C/08a16b8aa41a4f5da673c932b5a8b41c 2023-04-25 02:55:01,581 DEBUG [StoreCloser-TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C/aa98107974264ee8b9ef7be771d8e5e5 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C/aa98107974264ee8b9ef7be771d8e5e5 2023-04-25 02:55:01,582 DEBUG [StoreCloser-TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C/c8591ba6d2734211b1962f3142544a13 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C/c8591ba6d2734211b1962f3142544a13 2023-04-25 02:55:01,583 DEBUG [StoreCloser-TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C/acc14600f8304afd87bda8566ab499eb to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C/acc14600f8304afd87bda8566ab499eb 2023-04-25 02:55:01,583 DEBUG [StoreCloser-TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C/17c8516514d84ca69b3424137388e781 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C/17c8516514d84ca69b3424137388e781 2023-04-25 02:55:01,584 DEBUG [StoreCloser-TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C/cd8f31c2e6a94246a6a3dc333e52e680 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C/cd8f31c2e6a94246a6a3dc333e52e680 2023-04-25 02:55:01,585 DEBUG [StoreCloser-TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C/5df0c28533b544ecaac582e943da5814 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C/5df0c28533b544ecaac582e943da5814 2023-04-25 02:55:01,586 DEBUG [StoreCloser-TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C/0dee40da444048a098e3730c0e1f7e28 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C/0dee40da444048a098e3730c0e1f7e28 2023-04-25 02:55:01,587 DEBUG [StoreCloser-TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C/7c2a1f49b23a4fe08ab8c2567609d5aa to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C/7c2a1f49b23a4fe08ab8c2567609d5aa 2023-04-25 02:55:01,588 DEBUG [StoreCloser-TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C/5c76b8d71c534b07946f1548b5df5a4f to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C/5c76b8d71c534b07946f1548b5df5a4f 2023-04-25 02:55:01,588 DEBUG [StoreCloser-TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C/a53a6bec412a44d2aaeffb999e63b6a3 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C/a53a6bec412a44d2aaeffb999e63b6a3 2023-04-25 02:55:01,589 DEBUG [StoreCloser-TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C/ca9b6e0a629f4db186e5b8a47293b8fe to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C/ca9b6e0a629f4db186e5b8a47293b8fe 2023-04-25 02:55:01,590 DEBUG [StoreCloser-TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C/b06f0f1632084ffda1167091ff512f6e to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C/b06f0f1632084ffda1167091ff512f6e 2023-04-25 02:55:01,591 DEBUG [StoreCloser-TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C/2388ec51e2ea4ddd804098b60f33c29c to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C/2388ec51e2ea4ddd804098b60f33c29c 2023-04-25 02:55:01,592 DEBUG [StoreCloser-TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C/f78bd380cc2c4215b6527e71f3259150 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C/f78bd380cc2c4215b6527e71f3259150 2023-04-25 02:55:01,593 DEBUG [StoreCloser-TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C/37f186c853bb4d1bbc4c5996b10e31c2 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C/37f186c853bb4d1bbc4c5996b10e31c2 2023-04-25 02:55:01,593 DEBUG [StoreCloser-TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C/4a82ea8131454324850d359e769b09f5 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C/4a82ea8131454324850d359e769b09f5 2023-04-25 02:55:01,594 DEBUG [StoreCloser-TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C/e94e9f097d5448a59f5dfa2babd43a9d to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C/e94e9f097d5448a59f5dfa2babd43a9d 2023-04-25 02:55:01,595 DEBUG [StoreCloser-TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C/a2670746c44e4d2aa18000572b6b732c to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C/a2670746c44e4d2aa18000572b6b732c 2023-04-25 02:55:01,596 DEBUG [StoreCloser-TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C/83468ec498424ed29d394f7e69152196 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C/83468ec498424ed29d394f7e69152196 2023-04-25 02:55:01,597 DEBUG [StoreCloser-TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C/233e6ff660194a589437949a50fbce99 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C/233e6ff660194a589437949a50fbce99 2023-04-25 02:55:01,601 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/recovered.edits/1007.seqid, newMaxSeqId=1007, maxSeqId=1 2023-04-25 02:55:01,602 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b. 2023-04-25 02:55:01,602 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 40cf532a35b33bde962f4f28e36e887b: 2023-04-25 02:55:01,603 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 40cf532a35b33bde962f4f28e36e887b 2023-04-25 02:55:01,604 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=32 updating hbase:meta row=40cf532a35b33bde962f4f28e36e887b, regionState=CLOSED 2023-04-25 02:55:01,604 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682391301603"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682391301603"}]},"ts":"1682391301603"} 2023-04-25 02:55:01,605 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=33, resume processing ppid=32 2023-04-25 02:55:01,605 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=33, ppid=32, state=SUCCESS; CloseRegionProcedure 40cf532a35b33bde962f4f28e36e887b, server=jenkins-hbase4.apache.org,46697,1682391220023 in 2.0010 sec 2023-04-25 02:55:01,607 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=32, resume processing ppid=31 2023-04-25 02:55:01,607 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=32, ppid=31, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=40cf532a35b33bde962f4f28e36e887b, UNASSIGN in 2.0050 sec 2023-04-25 02:55:01,607 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682391301607"}]},"ts":"1682391301607"} 2023-04-25 02:55:01,608 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-25 02:55:01,609 INFO [PEWorker-3] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-25 02:55:01,610 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=31, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 2.0160 sec 2023-04-25 02:55:01,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-25 02:55:01,701 INFO [Listener at localhost/41915] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 31 completed 2023-04-25 02:55:01,701 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-04-25 02:55:01,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure2.ProcedureExecutor(1029): Stored pid=34, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-25 02:55:01,704 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=34, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-25 02:55:01,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1227): Checking to see if procedure is done pid=34 2023-04-25 02:55:01,704 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=34, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-25 02:55:01,709 DEBUG [HFileArchiver-7] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/.tmp/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b 2023-04-25 02:55:01,711 DEBUG [HFileArchiver-7] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/.tmp/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A, FileablePath, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/.tmp/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B, FileablePath, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/.tmp/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C, FileablePath, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/.tmp/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/recovered.edits] 2023-04-25 02:55:01,713 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/.tmp/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A/95e273d3596b4b6e988ccbcfa43f5aaa to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A/95e273d3596b4b6e988ccbcfa43f5aaa 2023-04-25 02:55:01,714 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/.tmp/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A/d6521da165cc47ff8075e84093423798 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A/d6521da165cc47ff8075e84093423798 2023-04-25 02:55:01,716 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/.tmp/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A/f3a397e14602453e919af8abccc4287c to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/A/f3a397e14602453e919af8abccc4287c 2023-04-25 02:55:01,718 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/.tmp/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B/8d7d6a55b5214cb9acd319835380e98b to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B/8d7d6a55b5214cb9acd319835380e98b 2023-04-25 02:55:01,719 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/.tmp/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B/8ddef905bf0d47498cee33327836b443 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B/8ddef905bf0d47498cee33327836b443 2023-04-25 02:55:01,720 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/.tmp/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B/bb311720ce02415aa852c1671edc2c00 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/B/bb311720ce02415aa852c1671edc2c00 2023-04-25 02:55:01,722 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/.tmp/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C/ab571f1cfe5a4e788e3e6b25a951b61a to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C/ab571f1cfe5a4e788e3e6b25a951b61a 2023-04-25 02:55:01,723 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/.tmp/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C/c33192b45bc4488b828676928d7d4cdb to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C/c33192b45bc4488b828676928d7d4cdb 2023-04-25 02:55:01,724 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/.tmp/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C/e694a6fddbce4541b859647259741f4f to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/C/e694a6fddbce4541b859647259741f4f 2023-04-25 02:55:01,727 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/.tmp/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/recovered.edits/1007.seqid to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b/recovered.edits/1007.seqid 2023-04-25 02:55:01,727 DEBUG [HFileArchiver-7] backup.HFileArchiver(596): Deleted hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/.tmp/data/default/TestAcidGuarantees/40cf532a35b33bde962f4f28e36e887b 2023-04-25 02:55:01,727 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-25 02:55:01,729 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=34, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-25 02:55:01,730 WARN [PEWorker-1] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-25 02:55:01,731 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-25 02:55:01,732 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=34, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-25 02:55:01,732 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-25 02:55:01,732 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1682391301732"}]},"ts":"9223372036854775807"} 2023-04-25 02:55:01,734 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-25 02:55:01,734 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 40cf532a35b33bde962f4f28e36e887b, NAME => 'TestAcidGuarantees,,1682391275522.40cf532a35b33bde962f4f28e36e887b.', STARTKEY => '', ENDKEY => ''}] 2023-04-25 02:55:01,734 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-25 02:55:01,734 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1682391301734"}]},"ts":"9223372036854775807"} 2023-04-25 02:55:01,735 INFO [PEWorker-1] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-25 02:55:01,736 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(130): Finished pid=34, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-25 02:55:01,737 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=34, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 35 msec 2023-04-25 02:55:01,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1227): Checking to see if procedure is done pid=34 2023-04-25 02:55:01,805 INFO [Listener at localhost/41915] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 34 completed 2023-04-25 02:55:01,817 INFO [Listener at localhost/41915] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testGetAtomicity Thread=229 (was 238), OpenFileDescriptor=576 (was 559) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=445 (was 500), ProcessCount=170 (was 170), AvailableMemoryMB=4258 (was 4305) 2023-04-25 02:55:01,829 INFO [Listener at localhost/41915] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMobScanAtomicity Thread=229, OpenFileDescriptor=576, MaxFileDescriptor=60000, SystemLoadAverage=445, ProcessCount=170, AvailableMemoryMB=4257 2023-04-25 02:55:01,831 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-25 02:55:01,831 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-25 02:55:01,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure2.ProcedureExecutor(1029): Stored pid=35, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-25 02:55:01,833 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-25 02:55:01,833 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 35 2023-04-25 02:55:01,834 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-25 02:55:01,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-25 02:55:01,835 DEBUG [HFileArchiver-8] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/.tmp/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5 2023-04-25 02:55:01,835 DEBUG [HFileArchiver-8] backup.HFileArchiver(153): Directory hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/.tmp/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5 empty. 2023-04-25 02:55:01,836 DEBUG [HFileArchiver-8] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/.tmp/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5 2023-04-25 02:55:01,836 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-25 02:55:01,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-25 02:55:02,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-25 02:55:02,243 DEBUG [PEWorker-5] util.FSTableDescriptors(570): Wrote into hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-25 02:55:02,245 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 9d5b411b5fe67e5bb007f4d36e3e11e5, NAME => 'TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/.tmp 2023-04-25 02:55:02,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-25 02:55:02,650 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-25 02:55:02,650 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 9d5b411b5fe67e5bb007f4d36e3e11e5, disabling compactions & flushes 2023-04-25 02:55:02,650 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. 2023-04-25 02:55:02,650 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. 2023-04-25 02:55:02,650 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. after waiting 0 ms 2023-04-25 02:55:02,650 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. 2023-04-25 02:55:02,650 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. 2023-04-25 02:55:02,650 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 9d5b411b5fe67e5bb007f4d36e3e11e5: 2023-04-25 02:55:02,653 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-25 02:55:02,653 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682391302653"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682391302653"}]},"ts":"1682391302653"} 2023-04-25 02:55:02,654 INFO [PEWorker-5] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-25 02:55:02,655 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-25 02:55:02,655 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682391302655"}]},"ts":"1682391302655"} 2023-04-25 02:55:02,656 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-25 02:55:02,659 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=9d5b411b5fe67e5bb007f4d36e3e11e5, ASSIGN}] 2023-04-25 02:55:02,660 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=9d5b411b5fe67e5bb007f4d36e3e11e5, ASSIGN 2023-04-25 02:55:02,661 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=9d5b411b5fe67e5bb007f4d36e3e11e5, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,46697,1682391220023; forceNewPlan=false, retain=false 2023-04-25 02:55:02,812 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=36 updating hbase:meta row=9d5b411b5fe67e5bb007f4d36e3e11e5, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:02,812 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682391302812"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682391302812"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682391302812"}]},"ts":"1682391302812"} 2023-04-25 02:55:02,814 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=37, ppid=36, state=RUNNABLE; OpenRegionProcedure 9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023}] 2023-04-25 02:55:02,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-25 02:55:02,968 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. 2023-04-25 02:55:02,968 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 9d5b411b5fe67e5bb007f4d36e3e11e5, NAME => 'TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5.', STARTKEY => '', ENDKEY => ''} 2023-04-25 02:55:02,969 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 9d5b411b5fe67e5bb007f4d36e3e11e5 2023-04-25 02:55:02,969 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-25 02:55:02,969 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 9d5b411b5fe67e5bb007f4d36e3e11e5 2023-04-25 02:55:02,969 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 9d5b411b5fe67e5bb007f4d36e3e11e5 2023-04-25 02:55:02,970 INFO [StoreOpener-9d5b411b5fe67e5bb007f4d36e3e11e5-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 9d5b411b5fe67e5bb007f4d36e3e11e5 2023-04-25 02:55:02,971 DEBUG [StoreOpener-9d5b411b5fe67e5bb007f4d36e3e11e5-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/A 2023-04-25 02:55:02,971 DEBUG [StoreOpener-9d5b411b5fe67e5bb007f4d36e3e11e5-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/A 2023-04-25 02:55:02,971 INFO [StoreOpener-9d5b411b5fe67e5bb007f4d36e3e11e5-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-25 02:55:02,971 INFO [StoreOpener-9d5b411b5fe67e5bb007f4d36e3e11e5-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 9d5b411b5fe67e5bb007f4d36e3e11e5 columnFamilyName A 2023-04-25 02:55:02,972 INFO [StoreOpener-9d5b411b5fe67e5bb007f4d36e3e11e5-1] regionserver.HStore(310): Store=9d5b411b5fe67e5bb007f4d36e3e11e5/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-25 02:55:02,972 INFO [StoreOpener-9d5b411b5fe67e5bb007f4d36e3e11e5-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 9d5b411b5fe67e5bb007f4d36e3e11e5 2023-04-25 02:55:02,973 DEBUG [StoreOpener-9d5b411b5fe67e5bb007f4d36e3e11e5-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/B 2023-04-25 02:55:02,973 DEBUG [StoreOpener-9d5b411b5fe67e5bb007f4d36e3e11e5-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/B 2023-04-25 02:55:02,973 INFO [StoreOpener-9d5b411b5fe67e5bb007f4d36e3e11e5-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-25 02:55:02,973 INFO [StoreOpener-9d5b411b5fe67e5bb007f4d36e3e11e5-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 9d5b411b5fe67e5bb007f4d36e3e11e5 columnFamilyName B 2023-04-25 02:55:02,974 INFO [StoreOpener-9d5b411b5fe67e5bb007f4d36e3e11e5-1] regionserver.HStore(310): Store=9d5b411b5fe67e5bb007f4d36e3e11e5/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-25 02:55:02,974 INFO [StoreOpener-9d5b411b5fe67e5bb007f4d36e3e11e5-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 9d5b411b5fe67e5bb007f4d36e3e11e5 2023-04-25 02:55:02,975 DEBUG [StoreOpener-9d5b411b5fe67e5bb007f4d36e3e11e5-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/C 2023-04-25 02:55:02,975 DEBUG [StoreOpener-9d5b411b5fe67e5bb007f4d36e3e11e5-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/C 2023-04-25 02:55:02,975 INFO [StoreOpener-9d5b411b5fe67e5bb007f4d36e3e11e5-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-25 02:55:02,976 INFO [StoreOpener-9d5b411b5fe67e5bb007f4d36e3e11e5-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 9d5b411b5fe67e5bb007f4d36e3e11e5 columnFamilyName C 2023-04-25 02:55:02,976 INFO [StoreOpener-9d5b411b5fe67e5bb007f4d36e3e11e5-1] regionserver.HStore(310): Store=9d5b411b5fe67e5bb007f4d36e3e11e5/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-25 02:55:02,976 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. 2023-04-25 02:55:02,977 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5 2023-04-25 02:55:02,977 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5 2023-04-25 02:55:02,979 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-25 02:55:02,980 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 9d5b411b5fe67e5bb007f4d36e3e11e5 2023-04-25 02:55:02,982 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-25 02:55:02,982 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 9d5b411b5fe67e5bb007f4d36e3e11e5; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=75465920, jitterRate=0.12452983856201172}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-25 02:55:02,982 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 9d5b411b5fe67e5bb007f4d36e3e11e5: 2023-04-25 02:55:02,983 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5., pid=37, masterSystemTime=1682391302965 2023-04-25 02:55:02,984 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. 2023-04-25 02:55:02,984 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. 2023-04-25 02:55:02,985 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=36 updating hbase:meta row=9d5b411b5fe67e5bb007f4d36e3e11e5, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:02,985 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682391302985"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1682391302985"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1682391302985"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1682391302985"}]},"ts":"1682391302985"} 2023-04-25 02:55:02,987 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=37, resume processing ppid=36 2023-04-25 02:55:02,987 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=37, ppid=36, state=SUCCESS; OpenRegionProcedure 9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 in 172 msec 2023-04-25 02:55:02,988 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=36, resume processing ppid=35 2023-04-25 02:55:02,988 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=36, ppid=35, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=9d5b411b5fe67e5bb007f4d36e3e11e5, ASSIGN in 328 msec 2023-04-25 02:55:02,989 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-25 02:55:02,989 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682391302989"}]},"ts":"1682391302989"} 2023-04-25 02:55:02,990 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-25 02:55:02,991 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-25 02:55:02,992 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=35, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1600 sec 2023-04-25 02:55:03,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-25 02:55:03,938 INFO [Listener at localhost/41915] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 35 completed 2023-04-25 02:55:03,939 DEBUG [Listener at localhost/41915] zookeeper.ReadOnlyZKClient(139): Connect 0x38341134 to 127.0.0.1:62054 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-25 02:55:03,944 DEBUG [Listener at localhost/41915] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4d010713, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-25 02:55:03,945 DEBUG [Listener at localhost/41915] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-25 02:55:03,947 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:54868, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-25 02:55:03,948 DEBUG [Listener at localhost/41915] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-25 02:55:03,949 INFO [RS-EventLoopGroup-1-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:58896, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-25 02:55:03,950 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-25 02:55:03,951 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.HMaster$12(2454): Client=jenkins//172.31.14.131 modify table TestAcidGuarantees from 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} to 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', MOB_THRESHOLD => '4', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-25 02:55:03,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure2.ProcedureExecutor(1029): Stored pid=38, state=RUNNABLE:MODIFY_TABLE_PREPARE; ModifyTableProcedure table=TestAcidGuarantees 2023-04-25 02:55:04,360 DEBUG [PEWorker-4] util.FSTableDescriptors(570): Wrote into hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-25 02:55:04,361 DEBUG [PEWorker-4] util.FSTableDescriptors(519): Deleted hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-25 02:55:04,361 INFO [PEWorker-4] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-25 02:55:04,363 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=39, ppid=38, state=RUNNABLE:REOPEN_TABLE_REGIONS_GET_REGIONS; ReopenTableRegionsProcedure table=TestAcidGuarantees}] 2023-04-25 02:55:04,365 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=9d5b411b5fe67e5bb007f4d36e3e11e5, REOPEN/MOVE}] 2023-04-25 02:55:04,366 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=9d5b411b5fe67e5bb007f4d36e3e11e5, REOPEN/MOVE 2023-04-25 02:55:04,366 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=9d5b411b5fe67e5bb007f4d36e3e11e5, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:04,367 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682391304366"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682391304366"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682391304366"}]},"ts":"1682391304366"} 2023-04-25 02:55:04,368 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=41, ppid=40, state=RUNNABLE; CloseRegionProcedure 9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023}] 2023-04-25 02:55:04,518 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 9d5b411b5fe67e5bb007f4d36e3e11e5 2023-04-25 02:55:04,519 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 9d5b411b5fe67e5bb007f4d36e3e11e5, disabling compactions & flushes 2023-04-25 02:55:04,519 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. 2023-04-25 02:55:04,519 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. 2023-04-25 02:55:04,519 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. after waiting 0 ms 2023-04-25 02:55:04,519 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. 2023-04-25 02:55:04,524 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-04-25 02:55:04,524 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. 2023-04-25 02:55:04,524 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 9d5b411b5fe67e5bb007f4d36e3e11e5: 2023-04-25 02:55:04,524 WARN [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(3509): Not adding moved region record: 9d5b411b5fe67e5bb007f4d36e3e11e5 to self. 2023-04-25 02:55:04,526 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 9d5b411b5fe67e5bb007f4d36e3e11e5 2023-04-25 02:55:04,526 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=9d5b411b5fe67e5bb007f4d36e3e11e5, regionState=CLOSED 2023-04-25 02:55:04,526 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682391304526"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682391304526"}]},"ts":"1682391304526"} 2023-04-25 02:55:04,528 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=41, resume processing ppid=40 2023-04-25 02:55:04,528 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=41, ppid=40, state=SUCCESS; CloseRegionProcedure 9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 in 160 msec 2023-04-25 02:55:04,529 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=9d5b411b5fe67e5bb007f4d36e3e11e5, REOPEN/MOVE; state=CLOSED, location=jenkins-hbase4.apache.org,46697,1682391220023; forceNewPlan=false, retain=true 2023-04-25 02:55:04,679 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=9d5b411b5fe67e5bb007f4d36e3e11e5, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:04,679 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682391304679"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682391304679"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682391304679"}]},"ts":"1682391304679"} 2023-04-25 02:55:04,680 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=42, ppid=40, state=RUNNABLE; OpenRegionProcedure 9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023}] 2023-04-25 02:55:04,835 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. 2023-04-25 02:55:04,836 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 9d5b411b5fe67e5bb007f4d36e3e11e5, NAME => 'TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5.', STARTKEY => '', ENDKEY => ''} 2023-04-25 02:55:04,836 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 9d5b411b5fe67e5bb007f4d36e3e11e5 2023-04-25 02:55:04,836 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-25 02:55:04,837 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 9d5b411b5fe67e5bb007f4d36e3e11e5 2023-04-25 02:55:04,837 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 9d5b411b5fe67e5bb007f4d36e3e11e5 2023-04-25 02:55:04,838 INFO [StoreOpener-9d5b411b5fe67e5bb007f4d36e3e11e5-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 9d5b411b5fe67e5bb007f4d36e3e11e5 2023-04-25 02:55:04,838 DEBUG [StoreOpener-9d5b411b5fe67e5bb007f4d36e3e11e5-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/A 2023-04-25 02:55:04,838 DEBUG [StoreOpener-9d5b411b5fe67e5bb007f4d36e3e11e5-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/A 2023-04-25 02:55:04,839 INFO [StoreOpener-9d5b411b5fe67e5bb007f4d36e3e11e5-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-25 02:55:04,839 INFO [StoreOpener-9d5b411b5fe67e5bb007f4d36e3e11e5-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 9d5b411b5fe67e5bb007f4d36e3e11e5 columnFamilyName A 2023-04-25 02:55:04,841 INFO [StoreOpener-9d5b411b5fe67e5bb007f4d36e3e11e5-1] regionserver.HStore(310): Store=9d5b411b5fe67e5bb007f4d36e3e11e5/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-25 02:55:04,841 INFO [StoreOpener-9d5b411b5fe67e5bb007f4d36e3e11e5-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 9d5b411b5fe67e5bb007f4d36e3e11e5 2023-04-25 02:55:04,842 DEBUG [StoreOpener-9d5b411b5fe67e5bb007f4d36e3e11e5-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/B 2023-04-25 02:55:04,842 DEBUG [StoreOpener-9d5b411b5fe67e5bb007f4d36e3e11e5-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/B 2023-04-25 02:55:04,842 INFO [StoreOpener-9d5b411b5fe67e5bb007f4d36e3e11e5-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-25 02:55:04,842 INFO [StoreOpener-9d5b411b5fe67e5bb007f4d36e3e11e5-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 9d5b411b5fe67e5bb007f4d36e3e11e5 columnFamilyName B 2023-04-25 02:55:04,843 INFO [StoreOpener-9d5b411b5fe67e5bb007f4d36e3e11e5-1] regionserver.HStore(310): Store=9d5b411b5fe67e5bb007f4d36e3e11e5/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-25 02:55:04,843 INFO [StoreOpener-9d5b411b5fe67e5bb007f4d36e3e11e5-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 9d5b411b5fe67e5bb007f4d36e3e11e5 2023-04-25 02:55:04,843 DEBUG [StoreOpener-9d5b411b5fe67e5bb007f4d36e3e11e5-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/C 2023-04-25 02:55:04,844 DEBUG [StoreOpener-9d5b411b5fe67e5bb007f4d36e3e11e5-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/C 2023-04-25 02:55:04,844 INFO [StoreOpener-9d5b411b5fe67e5bb007f4d36e3e11e5-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-25 02:55:04,844 INFO [StoreOpener-9d5b411b5fe67e5bb007f4d36e3e11e5-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 9d5b411b5fe67e5bb007f4d36e3e11e5 columnFamilyName C 2023-04-25 02:55:04,844 INFO [StoreOpener-9d5b411b5fe67e5bb007f4d36e3e11e5-1] regionserver.HStore(310): Store=9d5b411b5fe67e5bb007f4d36e3e11e5/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-25 02:55:04,844 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. 2023-04-25 02:55:04,845 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5 2023-04-25 02:55:04,846 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5 2023-04-25 02:55:04,848 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-25 02:55:04,849 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 9d5b411b5fe67e5bb007f4d36e3e11e5 2023-04-25 02:55:04,849 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 9d5b411b5fe67e5bb007f4d36e3e11e5; next sequenceid=5; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=70786545, jitterRate=0.05480171740055084}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-25 02:55:04,849 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 9d5b411b5fe67e5bb007f4d36e3e11e5: 2023-04-25 02:55:04,850 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5., pid=42, masterSystemTime=1682391304832 2023-04-25 02:55:04,851 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. 2023-04-25 02:55:04,851 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. 2023-04-25 02:55:04,851 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=9d5b411b5fe67e5bb007f4d36e3e11e5, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:04,851 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682391304851"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1682391304851"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1682391304851"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1682391304851"}]},"ts":"1682391304851"} 2023-04-25 02:55:04,853 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=42, resume processing ppid=40 2023-04-25 02:55:04,853 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=42, ppid=40, state=SUCCESS; OpenRegionProcedure 9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 in 172 msec 2023-04-25 02:55:04,854 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=40, resume processing ppid=39 2023-04-25 02:55:04,854 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=40, ppid=39, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=9d5b411b5fe67e5bb007f4d36e3e11e5, REOPEN/MOVE in 488 msec 2023-04-25 02:55:04,856 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=39, resume processing ppid=38 2023-04-25 02:55:04,856 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=39, ppid=38, state=SUCCESS; ReopenTableRegionsProcedure table=TestAcidGuarantees in 492 msec 2023-04-25 02:55:04,857 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=38, state=SUCCESS; ModifyTableProcedure table=TestAcidGuarantees in 905 msec 2023-04-25 02:55:04,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1227): Checking to see if procedure is done pid=38 2023-04-25 02:55:04,859 DEBUG [Listener at localhost/41915] zookeeper.ReadOnlyZKClient(139): Connect 0x71b5058b to 127.0.0.1:62054 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-25 02:55:04,863 DEBUG [Listener at localhost/41915] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@69100738, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-25 02:55:04,863 DEBUG [Listener at localhost/41915] zookeeper.ReadOnlyZKClient(139): Connect 0x3620ee35 to 127.0.0.1:62054 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-25 02:55:04,870 DEBUG [Listener at localhost/41915] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6f603c27, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-25 02:55:04,871 DEBUG [Listener at localhost/41915] zookeeper.ReadOnlyZKClient(139): Connect 0x4fe1db6a to 127.0.0.1:62054 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-25 02:55:04,874 DEBUG [Listener at localhost/41915] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7e01d5e5, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-25 02:55:04,875 DEBUG [Listener at localhost/41915] zookeeper.ReadOnlyZKClient(139): Connect 0x10377b50 to 127.0.0.1:62054 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-25 02:55:04,878 DEBUG [Listener at localhost/41915] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@51ccd8b7, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-25 02:55:04,878 DEBUG [Listener at localhost/41915] zookeeper.ReadOnlyZKClient(139): Connect 0x4366fcdd to 127.0.0.1:62054 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-25 02:55:04,882 DEBUG [Listener at localhost/41915] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@68a0e21a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-25 02:55:04,882 DEBUG [Listener at localhost/41915] zookeeper.ReadOnlyZKClient(139): Connect 0x1ec12b02 to 127.0.0.1:62054 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-25 02:55:04,886 DEBUG [Listener at localhost/41915] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@711ebaf2, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-25 02:55:04,886 DEBUG [Listener at localhost/41915] zookeeper.ReadOnlyZKClient(139): Connect 0x1d7ba242 to 127.0.0.1:62054 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-25 02:55:04,889 DEBUG [Listener at localhost/41915] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@a7503d4, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-25 02:55:04,890 DEBUG [Listener at localhost/41915] zookeeper.ReadOnlyZKClient(139): Connect 0x3e079da1 to 127.0.0.1:62054 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-25 02:55:04,895 DEBUG [Listener at localhost/41915] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7eae4191, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-25 02:55:04,895 DEBUG [Listener at localhost/41915] zookeeper.ReadOnlyZKClient(139): Connect 0x159c5e0c to 127.0.0.1:62054 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-25 02:55:04,900 DEBUG [Listener at localhost/41915] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1774338b, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-25 02:55:04,900 DEBUG [Listener at localhost/41915] zookeeper.ReadOnlyZKClient(139): Connect 0x7bec116b to 127.0.0.1:62054 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-25 02:55:04,903 DEBUG [Listener at localhost/41915] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@cf75c24, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-25 02:55:04,908 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:55:04,912 DEBUG [hconnection-0x4bd6a535-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-25 02:55:04,912 DEBUG [hconnection-0x2014b46f-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-25 02:55:04,912 DEBUG [hconnection-0x128ba2ba-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-25 02:55:04,912 DEBUG [hconnection-0x53206f0a-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-25 02:55:04,913 DEBUG [hconnection-0x52d621c3-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-25 02:55:04,913 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:54884, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-25 02:55:04,913 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:54910, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-25 02:55:04,914 DEBUG [hconnection-0x113728db-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-25 02:55:04,916 DEBUG [hconnection-0xe532a38-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-25 02:55:04,916 DEBUG [hconnection-0x51b3d8db-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-25 02:55:04,917 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:55:04,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:55:04,918 DEBUG [hconnection-0x7ab402ce-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-25 02:55:04,918 DEBUG [hconnection-0x19666dc-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-25 02:55:04,918 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:55:04,919 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:04,919 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:54894, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-25 02:55:04,919 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:54886, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-25 02:55:04,920 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:54940, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-25 02:55:04,920 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:54924, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-25 02:55:04,920 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:54950, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-25 02:55:04,919 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:55:04,925 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:54978, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-25 02:55:04,926 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:55:04,928 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:54964, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-25 02:55:04,928 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:54966, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-25 02:55:04,928 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:04,928 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:04,932 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:04,932 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:04,932 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:04,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:04,934 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:04,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:04,934 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:55:04,935 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:04,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:55:04,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:04,939 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:55:04,939 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:55:04,939 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:55:04,939 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:55:04,939 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:04,939 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:55:04,940 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:55:04,940 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:55:04,940 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. 2023-04-25 02:55:04,940 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. started... 2023-04-25 02:55:04,941 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 9d5b411b5fe67e5bb007f4d36e3e11e5 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-04-25 02:55:04,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(9158): Flush requested on 9d5b411b5fe67e5bb007f4d36e3e11e5 2023-04-25 02:55:04,942 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. as already flushing 2023-04-25 02:55:04,942 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9d5b411b5fe67e5bb007f4d36e3e11e5, store=A 2023-04-25 02:55:04,942 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:55:04,942 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9d5b411b5fe67e5bb007f4d36e3e11e5, store=B 2023-04-25 02:55:04,942 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:55:04,942 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9d5b411b5fe67e5bb007f4d36e3e11e5, store=C 2023-04-25 02:55:04,942 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:55:04,962 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:04,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54886 deadline: 1682391364957, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:04,963 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:04,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54894 deadline: 1682391364958, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:04,963 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:04,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54940 deadline: 1682391364958, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:04,963 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682391304950, max timestamp=1682391304961Num uniques 859; 2023-04-25 02:55:04,963 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-25 02:55:04,963 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682391304950, max timestamp=1682391304961Num uniques 859; 2023-04-25 02:55:04,963 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-25 02:55:04,963 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682391304950, max timestamp=1682391304961Num uniques 859; 2023-04-25 02:55:04,963 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-25 02:55:04,992 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:04,992 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:04,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54924 deadline: 1682391364985, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:04,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54884 deadline: 1682391364986, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:05,069 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:05,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54886 deadline: 1682391365063, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:05,069 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:05,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54894 deadline: 1682391365064, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:05,069 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:05,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54940 deadline: 1682391365064, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:05,096 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:05,096 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54884 deadline: 1682391365093, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:05,096 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:05,096 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54924 deadline: 1682391365093, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:05,277 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:05,277 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:05,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54886 deadline: 1682391365270, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:05,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54894 deadline: 1682391365271, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:05,278 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:05,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54940 deadline: 1682391365271, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:05,299 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:05,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54884 deadline: 1682391365297, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:05,299 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:05,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54924 deadline: 1682391365297, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:05,377 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304256336f64ccdad40c48647833b279fc760 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304256336f64ccdad40c48647833b279fc760 2023-04-25 02:55:05,580 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:05,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54940 deadline: 1682391365579, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:05,584 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:05,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54894 deadline: 1682391365580, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:05,584 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:05,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54886 deadline: 1682391365580, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:05,602 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:05,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54884 deadline: 1682391365600, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:05,605 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:05,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54924 deadline: 1682391365602, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:05,783 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/.tmp/A/ddd35b6b0e81423c9de1386458364536 2023-04-25 02:55:06,084 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:06,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54940 deadline: 1682391366083, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:06,093 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:06,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54894 deadline: 1682391366086, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:06,093 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:06,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54886 deadline: 1682391366089, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:06,111 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:06,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54884 deadline: 1682391366104, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:06,112 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:06,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54924 deadline: 1682391366109, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:06,193 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=15 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/.tmp/B/1db215561a814ddda5f10046d1fa87c9 2023-04-25 02:55:06,604 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=15 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/.tmp/C/ae3ee0ab1b41449aaefda15e270fb0ae 2023-04-25 02:55:06,609 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/.tmp/A/ddd35b6b0e81423c9de1386458364536 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/A/ddd35b6b0e81423c9de1386458364536 2023-04-25 02:55:06,613 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/A/ddd35b6b0e81423c9de1386458364536, entries=150, sequenceid=15, filesize=25.1 K 2023-04-25 02:55:06,614 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/.tmp/B/1db215561a814ddda5f10046d1fa87c9 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/B/1db215561a814ddda5f10046d1fa87c9 2023-04-25 02:55:06,617 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/B/1db215561a814ddda5f10046d1fa87c9, entries=150, sequenceid=15, filesize=11.6 K 2023-04-25 02:55:06,618 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/.tmp/C/ae3ee0ab1b41449aaefda15e270fb0ae as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/C/ae3ee0ab1b41449aaefda15e270fb0ae 2023-04-25 02:55:06,625 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/C/ae3ee0ab1b41449aaefda15e270fb0ae, entries=150, sequenceid=15, filesize=11.6 K 2023-04-25 02:55:06,626 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 9d5b411b5fe67e5bb007f4d36e3e11e5 in 1685ms, sequenceid=15, compaction requested=false 2023-04-25 02:55:06,626 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9d5b411b5fe67e5bb007f4d36e3e11e5: 2023-04-25 02:55:06,626 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. 2023-04-25 02:55:06,626 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:55:06,626 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:55:06,626 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:06,626 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:55:06,626 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:55:06,628 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:06,628 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:06,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:06,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:06,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:06,628 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:06,628 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:55:06,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:06,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:06,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:06,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:06,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:06,629 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:55:06,630 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@40b6e615[Count = 0] remaining members to acquire global barrier 2023-04-25 02:55:06,630 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:55:06,630 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:06,631 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:06,631 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:06,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:06,631 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:55:06,631 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:55:06,631 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:55:06,631 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:06,631 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:55:06,632 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:06,632 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:55:06,632 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:06,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:06,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:06,632 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:55:06,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:06,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:06,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:06,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:06,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:06,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:06,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:06,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:06,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:55:06,634 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:55:06,634 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:55:06,634 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:55:06,634 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:55:06,634 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:55:06,634 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:55:06,637 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:06,637 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:06,637 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:06,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:06,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:06,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:06,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:06,637 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:06,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:06,637 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:06,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:06,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:06,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:06,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:06,638 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:06,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:06,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:06,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:06,640 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:06,640 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:55:06,640 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:55:06,640 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:06,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:06,640 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:06,640 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:55:06,641 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:55:06,640 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:06,641 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:06,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:06,640 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:06,641 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:06,641 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:06,641 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:06,641 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:06,641 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:06,641 DEBUG [Thread-2260] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:55:06,641 DEBUG [Thread-2260] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:55:06,741 DEBUG [Thread-2260] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:55:06,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:55:06,743 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:55:06,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:55:06,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:55:06,745 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:55:06,745 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:06,745 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:55:06,745 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:55:06,745 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:06,745 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:06,747 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:06,747 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:06,747 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:06,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:06,747 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:06,747 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:55:06,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:06,748 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:06,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:55:06,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:06,748 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:55:06,748 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:55:06,748 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:55:06,748 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:55:06,748 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:06,749 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:55:06,749 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:55:06,749 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:55:06,749 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. 2023-04-25 02:55:06,749 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. started... 2023-04-25 02:55:06,749 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 9d5b411b5fe67e5bb007f4d36e3e11e5 3/3 column families, dataSize=127.47 KB heapSize=365.06 KB 2023-04-25 02:55:06,749 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9d5b411b5fe67e5bb007f4d36e3e11e5, store=A 2023-04-25 02:55:06,749 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 02:55:06,749 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9d5b411b5fe67e5bb007f4d36e3e11e5, store=B 2023-04-25 02:55:06,749 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 02:55:06,749 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9d5b411b5fe67e5bb007f4d36e3e11e5, store=C 2023-04-25 02:55:06,750 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 02:55:07,096 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(9158): Flush requested on 9d5b411b5fe67e5bb007f4d36e3e11e5 2023-04-25 02:55:07,096 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. as already flushing 2023-04-25 02:55:07,116 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:07,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54894 deadline: 1682391367114, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:07,121 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:07,121 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54940 deadline: 1682391367115, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:07,121 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:07,121 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54886 deadline: 1682391367115, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:07,121 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:07,121 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54884 deadline: 1682391367117, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:07,122 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:07,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54924 deadline: 1682391367121, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:07,130 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-25 02:55:07,162 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230425697234a67fd644a7b97272ebc1a5839d to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425697234a67fd644a7b97272ebc1a5839d 2023-04-25 02:55:07,167 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=51, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/.tmp/A/7bf464a021a04faebc51627281a358d2 2023-04-25 02:55:07,226 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:07,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54894 deadline: 1682391367217, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:07,227 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:07,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54940 deadline: 1682391367222, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:07,227 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:07,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54886 deadline: 1682391367222, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:07,432 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:07,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54894 deadline: 1682391367427, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:07,432 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:07,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54886 deadline: 1682391367428, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:07,433 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:07,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54940 deadline: 1682391367429, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:07,577 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=51 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/.tmp/B/9c43d51c27cc4b3dabae028f220942cb 2023-04-25 02:55:07,737 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:07,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54894 deadline: 1682391367733, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:07,738 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:07,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54886 deadline: 1682391367733, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:07,739 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:07,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54940 deadline: 1682391367735, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:07,995 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=51 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/.tmp/C/767622ada2784abda2d8abd45232161d 2023-04-25 02:55:08,000 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/.tmp/A/7bf464a021a04faebc51627281a358d2 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/A/7bf464a021a04faebc51627281a358d2 2023-04-25 02:55:08,003 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/A/7bf464a021a04faebc51627281a358d2, entries=150, sequenceid=51, filesize=25.1 K 2023-04-25 02:55:08,004 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/.tmp/B/9c43d51c27cc4b3dabae028f220942cb as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/B/9c43d51c27cc4b3dabae028f220942cb 2023-04-25 02:55:08,012 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/B/9c43d51c27cc4b3dabae028f220942cb, entries=150, sequenceid=51, filesize=11.6 K 2023-04-25 02:55:08,013 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/.tmp/C/767622ada2784abda2d8abd45232161d as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/C/767622ada2784abda2d8abd45232161d 2023-04-25 02:55:08,018 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/C/767622ada2784abda2d8abd45232161d, entries=150, sequenceid=51, filesize=11.6 K 2023-04-25 02:55:08,019 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~365.02 KB/373776, currentSize=60.38 KB/61830 for 9d5b411b5fe67e5bb007f4d36e3e11e5 in 1270ms, sequenceid=51, compaction requested=false 2023-04-25 02:55:08,019 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9d5b411b5fe67e5bb007f4d36e3e11e5: 2023-04-25 02:55:08,019 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. 2023-04-25 02:55:08,019 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:55:08,019 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:55:08,019 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:08,020 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:55:08,020 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:55:08,031 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:08,031 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:08,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:08,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:08,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:08,032 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:08,032 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:55:08,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:08,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:08,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:08,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:08,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:08,033 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:55:08,033 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@54f87a62[Count = 0] remaining members to acquire global barrier 2023-04-25 02:55:08,033 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:55:08,033 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:08,035 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:08,035 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:08,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:08,035 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:55:08,035 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:55:08,035 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:55:08,035 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:08,035 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:55:08,036 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:08,036 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:55:08,036 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:08,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:08,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:08,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:08,036 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:55:08,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:08,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:08,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:08,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:08,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:08,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:08,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:08,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:55:08,039 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:55:08,039 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:55:08,039 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:55:08,039 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:55:08,039 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:55:08,039 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:55:08,040 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:08,040 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:08,040 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:08,040 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:08,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:08,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:08,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:08,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:08,040 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:08,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:08,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:08,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:08,041 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:08,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:08,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:08,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:08,042 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:08,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:08,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:08,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:08,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:08,044 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:08,044 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:08,044 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:08,044 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:08,044 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:55:08,044 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:55:08,044 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:08,044 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:08,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:08,044 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:08,044 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:08,044 DEBUG [Thread-2260] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:55:08,044 DEBUG [Thread-2260] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:55:08,044 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:08,044 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:08,044 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:08,044 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:08,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:08,144 DEBUG [Thread-2260] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:55:08,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:55:08,146 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:55:08,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:55:08,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:55:08,148 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:55:08,148 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:08,148 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:55:08,148 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:55:08,148 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:08,148 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:08,150 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:08,150 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:08,150 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:08,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:08,151 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:08,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:08,151 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:55:08,151 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:08,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:55:08,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:08,152 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:55:08,152 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:55:08,152 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:55:08,152 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:55:08,152 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:08,152 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:55:08,153 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:55:08,153 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:55:08,153 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. 2023-04-25 02:55:08,153 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. started... 2023-04-25 02:55:08,153 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 9d5b411b5fe67e5bb007f4d36e3e11e5 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-25 02:55:08,153 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9d5b411b5fe67e5bb007f4d36e3e11e5, store=A 2023-04-25 02:55:08,153 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:55:08,153 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9d5b411b5fe67e5bb007f4d36e3e11e5, store=B 2023-04-25 02:55:08,153 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:55:08,154 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9d5b411b5fe67e5bb007f4d36e3e11e5, store=C 2023-04-25 02:55:08,154 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:55:08,169 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230425a73d2819ec5f48a2ab99fc5fea77d477 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425a73d2819ec5f48a2ab99fc5fea77d477 2023-04-25 02:55:08,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(9158): Flush requested on 9d5b411b5fe67e5bb007f4d36e3e11e5 2023-04-25 02:55:08,242 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. as already flushing 2023-04-25 02:55:08,276 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682391308266, max timestamp=1682391308274Num uniques 859; 2023-04-25 02:55:08,276 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-25 02:55:08,276 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682391308266, max timestamp=1682391308274Num uniques 859; 2023-04-25 02:55:08,276 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682391308266, max timestamp=1682391308274Num uniques 859; 2023-04-25 02:55:08,276 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-25 02:55:08,276 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-25 02:55:08,314 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:08,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54886 deadline: 1682391368306, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:08,318 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:08,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54894 deadline: 1682391368307, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:08,318 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:08,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54940 deadline: 1682391368308, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:08,420 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:08,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54886 deadline: 1682391368415, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:08,424 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:08,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54894 deadline: 1682391368419, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:08,425 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:08,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54940 deadline: 1682391368419, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:08,574 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=63, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/.tmp/A/93a2909c726741f0b881413a2c5cf7a5 2023-04-25 02:55:08,628 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:08,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54886 deadline: 1682391368622, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:08,629 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:08,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54894 deadline: 1682391368625, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:08,629 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:08,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54940 deadline: 1682391368626, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:08,936 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:08,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54886 deadline: 1682391368930, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:08,936 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:08,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54894 deadline: 1682391368932, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:08,936 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:08,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54940 deadline: 1682391368932, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:08,989 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=63 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/.tmp/B/d9cc943a70804409856182741fa323fc 2023-04-25 02:55:08,998 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=63 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/.tmp/C/f79cc538748c4dd5bed99c0e0a922ef8 2023-04-25 02:55:09,003 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/.tmp/A/93a2909c726741f0b881413a2c5cf7a5 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/A/93a2909c726741f0b881413a2c5cf7a5 2023-04-25 02:55:09,007 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/A/93a2909c726741f0b881413a2c5cf7a5, entries=150, sequenceid=63, filesize=25.1 K 2023-04-25 02:55:09,008 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/.tmp/B/d9cc943a70804409856182741fa323fc as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/B/d9cc943a70804409856182741fa323fc 2023-04-25 02:55:09,014 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/B/d9cc943a70804409856182741fa323fc, entries=150, sequenceid=63, filesize=11.6 K 2023-04-25 02:55:09,015 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/.tmp/C/f79cc538748c4dd5bed99c0e0a922ef8 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/C/f79cc538748c4dd5bed99c0e0a922ef8 2023-04-25 02:55:09,023 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/C/f79cc538748c4dd5bed99c0e0a922ef8, entries=150, sequenceid=63, filesize=11.6 K 2023-04-25 02:55:09,024 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=120.76 KB/123660 for 9d5b411b5fe67e5bb007f4d36e3e11e5 in 871ms, sequenceid=63, compaction requested=true 2023-04-25 02:55:09,024 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9d5b411b5fe67e5bb007f4d36e3e11e5: 2023-04-25 02:55:09,024 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. 2023-04-25 02:55:09,024 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:55:09,024 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:55:09,024 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:09,024 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:55:09,024 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:55:09,026 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:09,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:09,026 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:09,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:09,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:09,026 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:09,026 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:55:09,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:09,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:09,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:09,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:09,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:09,027 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:55:09,028 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@74faf457[Count = 0] remaining members to acquire global barrier 2023-04-25 02:55:09,028 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:55:09,028 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:09,029 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:09,029 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:09,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:09,029 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:55:09,029 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:55:09,029 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:55:09,029 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:09,029 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:55:09,030 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:09,030 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:55:09,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:09,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:09,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:09,030 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:09,030 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:55:09,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:09,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:09,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:09,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:09,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:09,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:09,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:09,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:55:09,033 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:55:09,033 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:55:09,033 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:55:09,033 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:55:09,034 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:55:09,034 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:55:09,036 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:09,036 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:09,036 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:09,036 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:09,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:09,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:09,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:09,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:09,037 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:09,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:09,037 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:09,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:09,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:09,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:09,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:09,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:09,038 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:09,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:09,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:09,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:09,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:09,042 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:09,042 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:09,042 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:09,042 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:09,042 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:09,042 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:09,042 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:09,042 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:09,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:09,042 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:09,042 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:55:09,042 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:55:09,042 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:09,042 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:09,043 DEBUG [Thread-2260] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:55:09,043 DEBUG [Thread-2260] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:55:09,043 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:09,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:09,043 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:55:09,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(9158): Flush requested on 9d5b411b5fe67e5bb007f4d36e3e11e5 2023-04-25 02:55:09,141 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 9d5b411b5fe67e5bb007f4d36e3e11e5 3/3 column families, dataSize=127.47 KB heapSize=360.38 KB 2023-04-25 02:55:09,141 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9d5b411b5fe67e5bb007f4d36e3e11e5, store=A 2023-04-25 02:55:09,141 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 02:55:09,141 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9d5b411b5fe67e5bb007f4d36e3e11e5, store=B 2023-04-25 02:55:09,141 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 02:55:09,141 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9d5b411b5fe67e5bb007f4d36e3e11e5, store=C 2023-04-25 02:55:09,141 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 02:55:09,143 DEBUG [Thread-2260] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:55:09,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:55:09,147 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:55:09,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:55:09,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:55:09,149 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:55:09,149 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:09,149 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:55:09,149 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:55:09,149 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:09,149 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:09,150 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:09,150 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:09,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:09,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:09,151 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:09,151 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:09,152 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:09,152 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:55:09,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:55:09,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:09,152 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:55:09,152 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:55:09,152 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:55:09,152 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:55:09,152 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:09,152 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:55:09,153 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:55:09,153 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:55:09,153 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. 2023-04-25 02:55:09,153 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. started... 2023-04-25 02:55:09,153 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. as already flushing 2023-04-25 02:55:09,153 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. 2023-04-25 02:55:09,153 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:55:09,153 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:55:09,153 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:09,153 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:55:09,153 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:55:09,156 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:09,156 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:09,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:09,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:09,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:09,156 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:09,156 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:55:09,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:09,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:09,157 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304254a475f44d20547209efa13bd2c9aa83c to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304254a475f44d20547209efa13bd2c9aa83c 2023-04-25 02:55:09,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:09,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:09,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:09,160 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:55:09,160 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6e1c5bf[Count = 0] remaining members to acquire global barrier 2023-04-25 02:55:09,160 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:55:09,160 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:09,161 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:09,161 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:09,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:09,161 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:55:09,161 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:55:09,161 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:55:09,161 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:09,161 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:55:09,162 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:09,162 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:55:09,162 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:09,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:09,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:09,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:09,162 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:55:09,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:09,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:09,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:09,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:09,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:09,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:09,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:09,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:55:09,165 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:55:09,165 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:55:09,165 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:55:09,165 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:55:09,165 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:55:09,165 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:55:09,166 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:09,166 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:09,166 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:09,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:09,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:09,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:09,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:09,166 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:09,167 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:09,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:09,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:09,167 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:09,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:09,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:09,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:09,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:09,168 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:09,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:09,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:09,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:09,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:09,171 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:09,171 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:09,172 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:09,172 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:55:09,172 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:09,172 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:09,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:09,172 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:55:09,172 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:55:09,172 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:09,172 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:09,172 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:09,172 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:09,172 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:09,172 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:09,172 DEBUG [Thread-2260] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:55:09,172 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:09,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:09,172 DEBUG [Thread-2260] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:55:09,182 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:09,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54884 deadline: 1682391369171, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:09,192 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:09,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54924 deadline: 1682391369182, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:09,272 DEBUG [Thread-2260] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:55:09,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:55:09,274 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:55:09,276 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:55:09,276 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:55:09,276 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:55:09,276 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:09,276 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:55:09,276 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:55:09,276 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:09,276 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:09,277 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:09,277 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:09,277 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:09,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:09,278 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:09,278 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:55:09,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:09,278 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:09,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:55:09,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:09,278 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:55:09,278 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:55:09,279 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:55:09,279 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:55:09,279 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:09,279 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:55:09,280 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:55:09,280 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:55:09,280 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. 2023-04-25 02:55:09,280 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. started... 2023-04-25 02:55:09,280 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. as already flushing 2023-04-25 02:55:09,280 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. 2023-04-25 02:55:09,280 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:55:09,280 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:55:09,280 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:09,280 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:55:09,280 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:55:09,281 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:09,281 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:09,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:09,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:09,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:09,282 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:09,282 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:55:09,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:09,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:09,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:09,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:09,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:09,283 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:55:09,283 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:55:09,283 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@71100d38[Count = 0] remaining members to acquire global barrier 2023-04-25 02:55:09,283 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:09,284 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:09,284 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:09,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:09,284 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:55:09,284 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:55:09,284 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:55:09,285 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:09,285 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:55:09,287 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:09,287 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:55:09,287 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:09,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:09,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:09,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:09,287 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:55:09,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:09,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:09,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:09,288 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:09,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:09,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54884 deadline: 1682391369283, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:09,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:09,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:09,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:09,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:55:09,289 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:55:09,289 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:55:09,289 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:55:09,289 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:55:09,289 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:55:09,290 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:55:09,291 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:09,291 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:09,291 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:09,291 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:09,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:09,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:09,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:09,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:09,291 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:09,291 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:09,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:09,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:09,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:09,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:09,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:09,293 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:09,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:09,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:09,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:09,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:09,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:09,295 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:09,295 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:09,295 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:09,295 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:09,295 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:09,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:09,295 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:55:09,295 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:55:09,295 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:09,295 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:09,295 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:09,295 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:09,295 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:09,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:09,295 DEBUG [Thread-2260] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:55:09,295 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:09,295 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:55:09,295 DEBUG [Thread-2260] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:55:09,296 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:09,297 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:09,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54924 deadline: 1682391369293, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:09,396 DEBUG [Thread-2260] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:55:09,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:55:09,398 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:55:09,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:55:09,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:55:09,399 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:55:09,399 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:09,399 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:55:09,399 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:55:09,400 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:09,400 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:09,401 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:09,401 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:09,401 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:09,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:09,401 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:09,401 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:55:09,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:09,401 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:09,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:55:09,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:09,402 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:55:09,402 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:55:09,402 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:55:09,402 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:55:09,402 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:09,402 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:55:09,402 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:55:09,402 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:55:09,402 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. 2023-04-25 02:55:09,402 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. started... 2023-04-25 02:55:09,402 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. as already flushing 2023-04-25 02:55:09,403 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. 2023-04-25 02:55:09,403 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:55:09,403 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:55:09,403 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:09,403 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:55:09,403 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:55:09,405 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:09,405 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:09,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:09,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:09,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:09,405 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:09,405 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:55:09,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:09,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:09,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:09,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:09,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:09,406 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:55:09,406 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@662c4212[Count = 0] remaining members to acquire global barrier 2023-04-25 02:55:09,406 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:55:09,407 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:09,407 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:09,407 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:09,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:09,408 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:55:09,408 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:55:09,408 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:55:09,408 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:09,408 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:55:09,409 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:55:09,409 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:09,409 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:09,409 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:55:09,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:09,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:09,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:09,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:09,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:09,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:09,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:09,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:09,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:09,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:09,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:55:09,411 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:55:09,411 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:55:09,411 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:55:09,412 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:55:09,412 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:55:09,412 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:55:09,413 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:09,413 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:09,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:09,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:09,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:09,413 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:09,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:09,413 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:09,413 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:09,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:09,413 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:09,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:09,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:09,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:09,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:09,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:09,415 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:09,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:09,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:09,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:09,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:09,417 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:09,417 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:09,418 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:09,418 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:09,418 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:55:09,418 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:55:09,418 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:09,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:09,418 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:09,418 DEBUG [Thread-2260] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:55:09,418 DEBUG [Thread-2260] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:55:09,418 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:09,418 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:55:09,418 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:09,418 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:09,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:09,418 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:09,418 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:09,418 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:09,441 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:09,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54894 deadline: 1682391369438, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:09,442 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:09,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54940 deadline: 1682391369438, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:09,442 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:09,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54886 deadline: 1682391369440, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:09,492 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:09,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54884 deadline: 1682391369489, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:09,504 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:09,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54924 deadline: 1682391369499, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:09,518 DEBUG [Thread-2260] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:55:09,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:55:09,520 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:55:09,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:55:09,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:55:09,522 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:55:09,522 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:09,522 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:55:09,522 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:55:09,522 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:09,522 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:09,526 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:09,526 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:09,526 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:09,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:09,526 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:09,526 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:55:09,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:09,526 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:09,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:55:09,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:09,527 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:55:09,527 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:55:09,527 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:55:09,527 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:55:09,527 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:09,527 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:55:09,527 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:55:09,527 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:55:09,528 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. 2023-04-25 02:55:09,528 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. started... 2023-04-25 02:55:09,528 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. as already flushing 2023-04-25 02:55:09,528 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. 2023-04-25 02:55:09,528 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:55:09,528 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:55:09,528 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:09,528 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:55:09,528 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:55:09,529 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:09,529 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:09,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:09,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:09,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:09,530 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:09,530 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:55:09,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:09,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:09,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:09,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:09,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:09,531 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:55:09,531 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@45575acc[Count = 0] remaining members to acquire global barrier 2023-04-25 02:55:09,531 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:55:09,531 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:09,533 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:09,533 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:09,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:09,533 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:09,533 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:55:09,533 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:55:09,533 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:55:09,534 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:55:09,535 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:09,535 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:55:09,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:09,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:09,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:09,535 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:09,535 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:55:09,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:09,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:09,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:09,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:09,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:09,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:09,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:09,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:55:09,537 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:55:09,537 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:55:09,537 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:55:09,537 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:55:09,537 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:55:09,537 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:55:09,538 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:09,538 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:09,538 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:09,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:09,539 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:09,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:09,539 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:09,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:09,539 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:09,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:09,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:09,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:09,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:09,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:09,540 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:09,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:09,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:09,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:09,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:09,542 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:09,542 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:09,542 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:09,542 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:09,542 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:09,542 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:09,542 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:55:09,542 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:55:09,542 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:09,542 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:09,542 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:09,543 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:55:09,543 DEBUG [Thread-2260] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:55:09,543 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:09,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:09,542 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:09,543 DEBUG [Thread-2260] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:55:09,543 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:09,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:09,572 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=98, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/.tmp/A/255914514a5e4a768a16f509fcf81a08 2023-04-25 02:55:09,643 DEBUG [Thread-2260] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:55:09,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:55:09,645 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:55:09,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:55:09,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:55:09,647 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:55:09,647 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:09,647 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:55:09,647 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:55:09,647 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:09,647 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:09,649 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:09,649 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:09,650 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:09,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:09,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:09,650 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:09,650 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:55:09,650 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:09,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:55:09,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:09,651 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:55:09,656 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:55:09,656 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:55:09,656 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:55:09,656 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:09,656 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:55:09,660 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:55:09,660 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:55:09,660 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. 2023-04-25 02:55:09,660 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. started... 2023-04-25 02:55:09,660 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. as already flushing 2023-04-25 02:55:09,660 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. 2023-04-25 02:55:09,660 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:55:09,660 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:55:09,660 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:09,660 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:55:09,660 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:55:09,663 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:09,663 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:09,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:09,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:09,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:09,663 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:09,663 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:55:09,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:09,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:09,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:09,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:09,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:09,665 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:55:09,665 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4d93a3e4[Count = 0] remaining members to acquire global barrier 2023-04-25 02:55:09,665 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:55:09,665 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:09,666 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:09,667 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:09,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:09,667 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:55:09,667 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:55:09,667 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:55:09,667 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:09,667 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:55:09,668 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:09,668 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:55:09,668 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:09,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:09,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:09,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:09,668 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:55:09,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:09,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:09,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:09,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:09,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:09,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:09,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:09,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:55:09,672 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:55:09,672 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:55:09,672 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:55:09,672 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:55:09,672 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:55:09,672 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:55:09,679 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:09,679 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:09,679 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:09,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:09,680 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:09,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:09,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:09,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:09,680 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:09,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:09,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:09,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:09,681 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:09,681 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:09,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:09,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:09,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:09,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:09,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:09,684 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:09,684 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:09,684 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:09,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:09,684 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:09,684 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:09,684 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:09,684 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:09,684 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:09,684 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:09,684 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:09,684 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:09,685 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:55:09,685 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:55:09,685 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:55:09,685 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:09,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:09,685 DEBUG [Thread-2260] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:55:09,685 DEBUG [Thread-2260] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:55:09,785 DEBUG [Thread-2260] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:55:09,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:55:09,787 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:55:09,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:55:09,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:55:09,788 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:55:09,788 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:09,788 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:55:09,788 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:55:09,789 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:09,789 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:09,790 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:09,790 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:09,790 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:09,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:09,790 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:09,790 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:55:09,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:09,790 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:09,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:55:09,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:09,791 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:55:09,791 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:55:09,791 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:55:09,791 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:55:09,791 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:09,791 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:55:09,791 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:55:09,791 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:55:09,792 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. 2023-04-25 02:55:09,792 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. started... 2023-04-25 02:55:09,792 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. as already flushing 2023-04-25 02:55:09,792 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. 2023-04-25 02:55:09,792 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:55:09,792 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:55:09,792 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:09,792 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:55:09,792 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:55:09,793 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:09,793 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:09,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:09,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:09,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:09,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:09,794 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:09,794 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:55:09,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:09,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:09,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:09,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:09,796 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:55:09,796 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:55:09,796 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:09,796 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3c1b1ea3[Count = 0] remaining members to acquire global barrier 2023-04-25 02:55:09,797 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:09,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54884 deadline: 1682391369793, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:09,798 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:09,798 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:09,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:09,798 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:55:09,798 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:55:09,798 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:09,798 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:55:09,798 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:55:09,800 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:09,800 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:55:09,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:09,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:09,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:09,800 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:09,800 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:55:09,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:09,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:09,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:09,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:09,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:09,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:09,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:09,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:55:09,802 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:55:09,802 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:55:09,802 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:55:09,802 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:55:09,802 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:55:09,802 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:55:09,804 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:09,804 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:09,804 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:09,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:09,804 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:09,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:09,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:09,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:09,804 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:09,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:09,804 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:09,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:09,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:09,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:09,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:09,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:09,805 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:09,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:09,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:09,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:09,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:09,809 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:09,809 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:09,809 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:09,809 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:09,810 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:09,810 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:09,810 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:55:09,810 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:55:09,810 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:09,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:09,810 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:09,810 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:09,810 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:09,810 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:09,810 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:55:09,810 DEBUG [Thread-2260] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:55:09,810 DEBUG [Thread-2260] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:55:09,810 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:09,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:09,813 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:09,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54924 deadline: 1682391369806, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:09,910 DEBUG [Thread-2260] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:55:09,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:55:09,912 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:55:09,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:55:09,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:55:09,914 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:55:09,914 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:09,914 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:55:09,914 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:55:09,914 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:09,914 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:09,916 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:09,916 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:09,916 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:09,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:09,916 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:09,916 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:55:09,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:09,917 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:09,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:55:09,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:09,917 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:55:09,917 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:55:09,917 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:55:09,917 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:55:09,917 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:09,917 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:55:09,918 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:55:09,918 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:55:09,918 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. 2023-04-25 02:55:09,918 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. started... 2023-04-25 02:55:09,918 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. as already flushing 2023-04-25 02:55:09,918 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. 2023-04-25 02:55:09,918 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:55:09,918 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:55:09,918 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:09,918 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:55:09,918 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:55:09,922 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:09,922 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:09,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:09,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:09,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:09,922 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:09,922 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:55:09,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:09,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:09,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:09,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:09,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:09,924 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:55:09,924 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3823e7e0[Count = 0] remaining members to acquire global barrier 2023-04-25 02:55:09,924 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:55:09,925 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:09,925 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:09,925 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:09,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:09,926 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:55:09,926 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:55:09,926 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:55:09,926 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:09,926 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:55:09,928 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:09,928 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:55:09,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:09,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:09,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:09,928 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:09,928 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:55:09,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:09,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:09,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:09,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:09,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:09,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:09,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:09,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:55:09,930 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:55:09,930 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:55:09,930 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:55:09,930 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:55:09,930 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:55:09,930 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:55:09,931 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:09,931 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:09,931 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:09,931 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:09,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:09,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:09,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:09,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:09,932 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:09,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:09,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:09,932 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:09,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:09,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:09,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:09,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:09,933 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:09,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:09,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:09,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:09,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:09,935 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:09,935 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:09,935 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:09,936 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:09,936 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:09,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:09,935 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:09,936 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:09,936 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:09,936 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:09,936 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:09,936 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:09,936 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:55:09,936 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:55:09,936 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:09,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:09,936 DEBUG [Thread-2260] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:55:09,936 DEBUG [Thread-2260] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:55:09,983 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=98 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/.tmp/B/4f7a577a5df44d17bb388655eb7aff52 2023-04-25 02:55:10,036 DEBUG [Thread-2260] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:55:10,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:55:10,038 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:55:10,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:55:10,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:55:10,039 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:55:10,039 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:10,039 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:55:10,040 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:55:10,040 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:10,040 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:10,041 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:10,041 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:10,041 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:10,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:10,041 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:10,041 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:55:10,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:10,042 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:10,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:55:10,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:10,042 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:55:10,042 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:55:10,042 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:55:10,042 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:55:10,042 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:10,042 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:55:10,042 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:55:10,042 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:55:10,043 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. 2023-04-25 02:55:10,043 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. started... 2023-04-25 02:55:10,043 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. as already flushing 2023-04-25 02:55:10,043 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. 2023-04-25 02:55:10,043 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:55:10,043 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:55:10,043 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:10,043 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:55:10,043 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:55:10,045 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:10,045 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:10,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:10,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:10,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:10,045 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:10,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:10,045 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:55:10,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:10,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:10,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:10,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:10,047 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:55:10,047 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:55:10,047 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@475d4b12[Count = 0] remaining members to acquire global barrier 2023-04-25 02:55:10,047 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:10,048 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:10,048 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:10,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:10,049 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:55:10,049 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:55:10,049 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:55:10,049 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:10,049 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:55:10,050 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:55:10,050 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:10,050 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:10,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:10,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:10,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:10,050 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:55:10,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:10,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:10,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:10,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:10,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:10,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:10,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:10,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:55:10,052 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:55:10,052 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:55:10,052 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:55:10,052 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:55:10,052 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:55:10,052 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:55:10,054 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:10,054 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:10,054 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:10,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:10,054 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:10,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:10,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:10,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:10,054 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:10,054 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:10,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:10,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:10,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:10,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:10,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:10,055 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:10,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:10,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:10,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:10,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:10,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:10,058 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:10,058 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:10,058 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:55:10,058 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:55:10,058 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:10,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:10,058 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:10,059 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:55:10,058 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:10,058 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:10,059 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:10,059 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:10,059 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:10,059 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:10,059 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:10,059 DEBUG [Thread-2260] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:55:10,059 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:10,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:10,059 DEBUG [Thread-2260] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:55:10,159 DEBUG [Thread-2260] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:55:10,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:55:10,161 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:55:10,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:55:10,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:55:10,162 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:55:10,162 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:10,162 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:55:10,162 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:55:10,163 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:10,163 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:10,164 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:10,164 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:10,164 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:10,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:10,164 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:10,164 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:55:10,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:10,165 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:10,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:55:10,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:10,165 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:55:10,165 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:55:10,165 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:55:10,165 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:55:10,166 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:10,166 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:55:10,166 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:55:10,166 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:55:10,166 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. 2023-04-25 02:55:10,166 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. started... 2023-04-25 02:55:10,166 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. as already flushing 2023-04-25 02:55:10,166 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. 2023-04-25 02:55:10,166 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:55:10,166 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:55:10,166 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:10,166 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:55:10,166 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:55:10,168 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:10,168 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:10,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:10,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:10,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:10,168 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:10,168 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:55:10,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:10,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:10,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:10,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:10,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:10,169 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:55:10,169 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@17014c9e[Count = 0] remaining members to acquire global barrier 2023-04-25 02:55:10,169 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:55:10,169 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:10,170 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:10,170 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:10,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:10,170 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:55:10,170 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:55:10,170 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:55:10,170 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:10,171 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:55:10,173 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:10,173 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:55:10,173 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:10,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:10,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:10,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:10,173 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:55:10,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:10,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:10,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:10,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:10,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:10,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:10,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:10,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:55:10,175 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:55:10,175 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:55:10,175 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:55:10,175 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:55:10,175 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:55:10,175 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:55:10,177 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:10,177 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:10,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:10,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:10,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:10,177 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:10,177 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:10,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:10,177 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:10,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:10,177 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:10,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:10,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:10,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:10,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:10,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:10,178 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:10,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:10,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:10,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:10,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:10,181 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:10,181 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:10,181 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:10,185 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:10,185 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:10,185 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:10,185 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:10,185 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:10,185 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:10,185 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:55:10,185 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:55:10,185 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:55:10,185 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:10,185 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:10,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:10,185 DEBUG [Thread-2260] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:55:10,185 DEBUG [Thread-2260] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:55:10,185 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:10,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:10,285 DEBUG [Thread-2260] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:55:10,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:55:10,287 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:55:10,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:55:10,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:55:10,289 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:55:10,289 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:10,289 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:55:10,289 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:55:10,289 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:10,289 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:10,290 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:10,290 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:10,290 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:10,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:10,291 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:10,291 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:55:10,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:10,291 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:10,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:55:10,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:10,292 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:55:10,292 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:55:10,292 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:55:10,292 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:55:10,292 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:10,292 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:55:10,292 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:55:10,292 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:55:10,292 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. 2023-04-25 02:55:10,292 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. started... 2023-04-25 02:55:10,292 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. as already flushing 2023-04-25 02:55:10,293 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. 2023-04-25 02:55:10,293 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:55:10,293 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:55:10,293 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:10,293 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:55:10,293 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:55:10,295 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:10,295 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:10,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:10,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:10,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:10,295 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:10,295 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:55:10,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:10,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:10,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:10,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:10,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:10,296 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:55:10,297 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@738bb6db[Count = 0] remaining members to acquire global barrier 2023-04-25 02:55:10,297 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:55:10,297 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:10,298 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:10,298 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:10,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:10,298 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:55:10,298 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:55:10,298 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:10,298 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:55:10,298 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:55:10,302 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:10,302 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:55:10,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:10,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:10,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:10,302 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:10,302 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:55:10,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:10,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:10,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:10,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:10,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:10,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:10,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:10,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:55:10,304 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:55:10,304 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:55:10,304 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:55:10,304 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:55:10,304 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:55:10,304 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:55:10,305 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:10,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54884 deadline: 1682391370300, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:10,306 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:10,306 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:10,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:10,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:10,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:10,306 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:10,306 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:10,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:10,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:10,306 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:10,306 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:10,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:10,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:10,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:10,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:10,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:10,307 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:10,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:10,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:10,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:10,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:10,309 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:10,310 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:10,310 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:10,310 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:10,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:10,310 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:10,310 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:55:10,310 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:55:10,310 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:10,310 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:10,310 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:10,310 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:10,310 DEBUG [Thread-2260] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:55:10,310 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:10,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:10,310 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:10,310 DEBUG [Thread-2260] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:55:10,310 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:10,321 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:10,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54924 deadline: 1682391370315, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:10,393 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=98 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/.tmp/C/824b7a6dac994061a7b5eec7dd1c393d 2023-04-25 02:55:10,397 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/.tmp/A/255914514a5e4a768a16f509fcf81a08 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/A/255914514a5e4a768a16f509fcf81a08 2023-04-25 02:55:10,401 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/A/255914514a5e4a768a16f509fcf81a08, entries=200, sequenceid=98, filesize=31.9 K 2023-04-25 02:55:10,402 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/.tmp/B/4f7a577a5df44d17bb388655eb7aff52 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/B/4f7a577a5df44d17bb388655eb7aff52 2023-04-25 02:55:10,405 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/B/4f7a577a5df44d17bb388655eb7aff52, entries=150, sequenceid=98, filesize=11.6 K 2023-04-25 02:55:10,406 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/.tmp/C/824b7a6dac994061a7b5eec7dd1c393d as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/C/824b7a6dac994061a7b5eec7dd1c393d 2023-04-25 02:55:10,409 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/C/824b7a6dac994061a7b5eec7dd1c393d, entries=150, sequenceid=98, filesize=11.6 K 2023-04-25 02:55:10,410 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~360.33 KB/368976, currentSize=53.67 KB/54960 for 9d5b411b5fe67e5bb007f4d36e3e11e5 in 1269ms, sequenceid=98, compaction requested=true 2023-04-25 02:55:10,410 DEBUG [Thread-2260] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:55:10,410 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 9d5b411b5fe67e5bb007f4d36e3e11e5: 2023-04-25 02:55:10,411 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 02:55:10,411 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-25 02:55:10,411 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-25 02:55:10,411 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 02:55:10,411 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 02:55:10,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:55:10,412 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 109924 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-25 02:55:10,413 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HStore(1912): 9d5b411b5fe67e5bb007f4d36e3e11e5/A is initiating minor compaction (all files) 2023-04-25 02:55:10,413 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 9d5b411b5fe67e5bb007f4d36e3e11e5/A in TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. 2023-04-25 02:55:10,413 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/A/ddd35b6b0e81423c9de1386458364536, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/A/7bf464a021a04faebc51627281a358d2, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/A/93a2909c726741f0b881413a2c5cf7a5, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/A/255914514a5e4a768a16f509fcf81a08] into tmpdir=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/.tmp, totalSize=107.3 K 2023-04-25 02:55:10,413 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting ddd35b6b0e81423c9de1386458364536, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=15, earliestPutTs=1682391304935 2023-04-25 02:55:10,414 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:55:10,414 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting 7bf464a021a04faebc51627281a358d2, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=51, earliestPutTs=1682391304966 2023-04-25 02:55:10,414 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47544 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-25 02:55:10,414 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HStore(1912): 9d5b411b5fe67e5bb007f4d36e3e11e5/B is initiating minor compaction (all files) 2023-04-25 02:55:10,414 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 9d5b411b5fe67e5bb007f4d36e3e11e5/B in TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. 2023-04-25 02:55:10,414 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/B/1db215561a814ddda5f10046d1fa87c9, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/B/9c43d51c27cc4b3dabae028f220942cb, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/B/d9cc943a70804409856182741fa323fc, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/B/4f7a577a5df44d17bb388655eb7aff52] into tmpdir=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/.tmp, totalSize=46.4 K 2023-04-25 02:55:10,415 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting 93a2909c726741f0b881413a2c5cf7a5, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=63, earliestPutTs=1682391307108 2023-04-25 02:55:10,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:55:10,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:55:10,416 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:55:10,416 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:10,416 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:55:10,416 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:55:10,417 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:10,417 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:10,417 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting 1db215561a814ddda5f10046d1fa87c9, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=15, earliestPutTs=1682391304935 2023-04-25 02:55:10,417 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting 255914514a5e4a768a16f509fcf81a08, keycount=200, bloomtype=ROW, size=31.9 K, encoding=NONE, compression=NONE, seqNum=98, earliestPutTs=1682391308305 2023-04-25 02:55:10,417 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting 9c43d51c27cc4b3dabae028f220942cb, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=51, earliestPutTs=1682391304966 2023-04-25 02:55:10,418 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting d9cc943a70804409856182741fa323fc, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=63, earliestPutTs=1682391307108 2023-04-25 02:55:10,418 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting 4f7a577a5df44d17bb388655eb7aff52, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=98, earliestPutTs=1682391308306 2023-04-25 02:55:10,419 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:10,419 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:10,419 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:10,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:10,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:10,420 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:10,420 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:55:10,420 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:10,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:55:10,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:10,421 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:55:10,424 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:55:10,424 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:55:10,424 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:55:10,424 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:10,424 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:55:10,428 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:55:10,428 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:55:10,428 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. 2023-04-25 02:55:10,428 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. started... 2023-04-25 02:55:10,428 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 9d5b411b5fe67e5bb007f4d36e3e11e5 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-25 02:55:10,428 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9d5b411b5fe67e5bb007f4d36e3e11e5, store=A 2023-04-25 02:55:10,428 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:55:10,429 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9d5b411b5fe67e5bb007f4d36e3e11e5, store=B 2023-04-25 02:55:10,429 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:55:10,429 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9d5b411b5fe67e5bb007f4d36e3e11e5, store=C 2023-04-25 02:55:10,429 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:55:10,433 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] throttle.PressureAwareThroughputController(145): 9d5b411b5fe67e5bb007f4d36e3e11e5#B#compaction#306 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-25 02:55:10,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(9158): Flush requested on 9d5b411b5fe67e5bb007f4d36e3e11e5 2023-04-25 02:55:10,446 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. as already flushing 2023-04-25 02:55:10,461 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] throttle.PressureAwareThroughputController(145): 9d5b411b5fe67e5bb007f4d36e3e11e5#A#compaction#307 average throughput is 0.90 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-25 02:55:10,474 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230425e6a540a70d9f4bfa92b4d436004f167f to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425e6a540a70d9f4bfa92b4d436004f167f 2023-04-25 02:55:10,485 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=109, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/.tmp/A/0819a75a91174ab38d49623fbe728436 2023-04-25 02:55:10,505 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682391310493, max timestamp=1682391310503Num uniques 859; 2023-04-25 02:55:10,505 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682391310493, max timestamp=1682391310503Num uniques 859; 2023-04-25 02:55:10,505 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-25 02:55:10,505 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682391310493, max timestamp=1682391310503Num uniques 859; 2023-04-25 02:55:10,505 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-25 02:55:10,505 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-25 02:55:10,564 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:10,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54894 deadline: 1682391370551, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:10,568 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:10,568 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54940 deadline: 1682391370560, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:10,569 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:10,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54886 deadline: 1682391370564, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:10,671 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:10,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54894 deadline: 1682391370665, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:10,671 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:10,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54940 deadline: 1682391370669, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:10,672 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:10,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54886 deadline: 1682391370669, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:10,865 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/.tmp/B/4452115a19bd4f5e83148150b3a27076 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/B/4452115a19bd4f5e83148150b3a27076 2023-04-25 02:55:10,869 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 9d5b411b5fe67e5bb007f4d36e3e11e5/B of 9d5b411b5fe67e5bb007f4d36e3e11e5 into 4452115a19bd4f5e83148150b3a27076(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 02:55:10,869 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9d5b411b5fe67e5bb007f4d36e3e11e5: 2023-04-25 02:55:10,869 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5., storeName=9d5b411b5fe67e5bb007f4d36e3e11e5/B, priority=12, startTime=1682391310411; duration=0sec 2023-04-25 02:55:10,870 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 02:55:10,870 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-25 02:55:10,871 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47544 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-25 02:55:10,871 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HStore(1912): 9d5b411b5fe67e5bb007f4d36e3e11e5/C is initiating minor compaction (all files) 2023-04-25 02:55:10,871 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 9d5b411b5fe67e5bb007f4d36e3e11e5/C in TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. 2023-04-25 02:55:10,871 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/C/ae3ee0ab1b41449aaefda15e270fb0ae, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/C/767622ada2784abda2d8abd45232161d, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/C/f79cc538748c4dd5bed99c0e0a922ef8, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/C/824b7a6dac994061a7b5eec7dd1c393d] into tmpdir=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/.tmp, totalSize=46.4 K 2023-04-25 02:55:10,872 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting ae3ee0ab1b41449aaefda15e270fb0ae, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=15, earliestPutTs=1682391304935 2023-04-25 02:55:10,872 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting 767622ada2784abda2d8abd45232161d, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=51, earliestPutTs=1682391304966 2023-04-25 02:55:10,873 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting f79cc538748c4dd5bed99c0e0a922ef8, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=63, earliestPutTs=1682391307108 2023-04-25 02:55:10,874 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting 824b7a6dac994061a7b5eec7dd1c393d, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=98, earliestPutTs=1682391308306 2023-04-25 02:55:10,878 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:10,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54894 deadline: 1682391370872, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:10,878 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:10,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54940 deadline: 1682391370873, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:10,878 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:10,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54886 deadline: 1682391370874, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:10,881 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] throttle.PressureAwareThroughputController(145): 9d5b411b5fe67e5bb007f4d36e3e11e5#C#compaction#310 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-25 02:55:10,913 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=109 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/.tmp/B/886195b72dee4da182fa19a6b333d181 2023-04-25 02:55:11,183 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:11,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54894 deadline: 1682391371180, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:11,183 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:11,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54886 deadline: 1682391371180, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:11,187 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:11,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54940 deadline: 1682391371181, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:11,291 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/.tmp/C/73446cf519cf44218916175b681de98f as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/C/73446cf519cf44218916175b681de98f 2023-04-25 02:55:11,296 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 9d5b411b5fe67e5bb007f4d36e3e11e5/C of 9d5b411b5fe67e5bb007f4d36e3e11e5 into 73446cf519cf44218916175b681de98f(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 02:55:11,296 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9d5b411b5fe67e5bb007f4d36e3e11e5: 2023-04-25 02:55:11,296 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5., storeName=9d5b411b5fe67e5bb007f4d36e3e11e5/C, priority=12, startTime=1682391310411; duration=0sec 2023-04-25 02:55:11,296 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 02:55:11,318 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:11,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54884 deadline: 1682391371313, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:11,323 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=109 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/.tmp/C/68d78dd5c2834816b8ca1a20c1cbe8b4 2023-04-25 02:55:11,327 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/.tmp/A/0819a75a91174ab38d49623fbe728436 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/A/0819a75a91174ab38d49623fbe728436 2023-04-25 02:55:11,328 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:11,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54924 deadline: 1682391371326, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:11,331 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/A/0819a75a91174ab38d49623fbe728436, entries=150, sequenceid=109, filesize=25.1 K 2023-04-25 02:55:11,331 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/.tmp/B/886195b72dee4da182fa19a6b333d181 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/B/886195b72dee4da182fa19a6b333d181 2023-04-25 02:55:11,335 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/B/886195b72dee4da182fa19a6b333d181, entries=150, sequenceid=109, filesize=11.6 K 2023-04-25 02:55:11,335 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/.tmp/C/68d78dd5c2834816b8ca1a20c1cbe8b4 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/C/68d78dd5c2834816b8ca1a20c1cbe8b4 2023-04-25 02:55:11,338 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/C/68d78dd5c2834816b8ca1a20c1cbe8b4, entries=150, sequenceid=109, filesize=11.6 K 2023-04-25 02:55:11,342 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 9d5b411b5fe67e5bb007f4d36e3e11e5 in 911ms, sequenceid=109, compaction requested=false 2023-04-25 02:55:11,342 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9d5b411b5fe67e5bb007f4d36e3e11e5: 2023-04-25 02:55:11,342 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. 2023-04-25 02:55:11,342 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:55:11,342 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:55:11,342 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:11,342 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:55:11,342 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:55:11,343 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:11,344 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:11,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:11,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:11,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:11,344 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:11,344 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:55:11,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:11,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:11,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:11,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:11,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:11,345 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:55:11,345 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@543053ad[Count = 0] remaining members to acquire global barrier 2023-04-25 02:55:11,345 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:55:11,346 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:11,346 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:11,346 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:11,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:11,347 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:55:11,347 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:55:11,347 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:55:11,347 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:11,347 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:55:11,348 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:55:11,348 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:11,348 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:11,348 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:55:11,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:11,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:11,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:11,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:11,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:11,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:11,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:11,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:11,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:11,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:11,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:55:11,350 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:55:11,350 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:55:11,350 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:55:11,350 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:55:11,350 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:55:11,350 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:55:11,353 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:11,353 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:11,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:11,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:11,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:11,353 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:11,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:11,353 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:11,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:11,354 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:11,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:11,353 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:11,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:11,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:11,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:11,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:11,355 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:11,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:11,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:11,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:11,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:11,357 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:11,357 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:11,357 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:11,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:11,357 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:11,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:11,359 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:11,359 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:11,359 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:11,359 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:11,359 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:55:11,359 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:11,359 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:55:11,359 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:55:11,359 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:11,359 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:11,360 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:11,360 DEBUG [Thread-2260] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:55:11,360 DEBUG [Thread-2260] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:55:11,460 DEBUG [Thread-2260] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:55:11,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:55:11,462 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:55:11,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:55:11,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:55:11,463 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:55:11,463 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:11,463 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:55:11,463 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:55:11,463 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:11,463 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:11,464 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:11,464 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:11,464 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:11,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:11,465 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:11,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:11,465 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:55:11,465 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:11,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:55:11,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:11,465 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:55:11,465 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:55:11,465 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:55:11,466 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:11,466 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:55:11,466 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:55:11,466 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:55:11,466 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. 2023-04-25 02:55:11,466 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. started... 2023-04-25 02:55:11,466 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 9d5b411b5fe67e5bb007f4d36e3e11e5 3/3 column families, dataSize=127.47 KB heapSize=360.38 KB 2023-04-25 02:55:11,466 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9d5b411b5fe67e5bb007f4d36e3e11e5, store=A 2023-04-25 02:55:11,466 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 02:55:11,466 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9d5b411b5fe67e5bb007f4d36e3e11e5, store=B 2023-04-25 02:55:11,466 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 02:55:11,466 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9d5b411b5fe67e5bb007f4d36e3e11e5, store=C 2023-04-25 02:55:11,466 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 02:55:11,483 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230425c53e91b5c1684163ba26c6914d561ff0 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425c53e91b5c1684163ba26c6914d561ff0 2023-04-25 02:55:11,489 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=146, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/.tmp/A/b82f6c1510c545ba8f5be887bb396bbe 2023-04-25 02:55:11,685 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/.tmp/A/fde37d724a4447e189b46f2fe3ded178 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/A/fde37d724a4447e189b46f2fe3ded178 2023-04-25 02:55:11,689 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 9d5b411b5fe67e5bb007f4d36e3e11e5/A of 9d5b411b5fe67e5bb007f4d36e3e11e5 into fde37d724a4447e189b46f2fe3ded178(size=25.3 K), total size for store is 50.4 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-04-25 02:55:11,689 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9d5b411b5fe67e5bb007f4d36e3e11e5: 2023-04-25 02:55:11,689 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5., storeName=9d5b411b5fe67e5bb007f4d36e3e11e5/A, priority=12, startTime=1682391310411; duration=1sec 2023-04-25 02:55:11,689 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 02:55:11,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(9158): Flush requested on 9d5b411b5fe67e5bb007f4d36e3e11e5 2023-04-25 02:55:11,692 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. as already flushing 2023-04-25 02:55:11,708 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:11,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54894 deadline: 1682391371703, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:11,714 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:11,714 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:11,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54886 deadline: 1682391371708, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:11,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54940 deadline: 1682391371708, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:11,812 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:11,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54894 deadline: 1682391371809, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:11,819 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:11,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54886 deadline: 1682391371815, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:11,819 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:11,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54940 deadline: 1682391371815, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:11,900 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=146 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/.tmp/B/765c196572fd4dc9999fca242f93b77b 2023-04-25 02:55:11,911 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=146 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/.tmp/C/1616975a2e1b498b82f75389ed6147a0 2023-04-25 02:55:11,915 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/.tmp/A/b82f6c1510c545ba8f5be887bb396bbe as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/A/b82f6c1510c545ba8f5be887bb396bbe 2023-04-25 02:55:11,918 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/A/b82f6c1510c545ba8f5be887bb396bbe, entries=150, sequenceid=146, filesize=25.3 K 2023-04-25 02:55:11,922 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/.tmp/B/765c196572fd4dc9999fca242f93b77b as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/B/765c196572fd4dc9999fca242f93b77b 2023-04-25 02:55:11,925 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/B/765c196572fd4dc9999fca242f93b77b, entries=150, sequenceid=146, filesize=11.8 K 2023-04-25 02:55:11,928 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/.tmp/C/1616975a2e1b498b82f75389ed6147a0 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/C/1616975a2e1b498b82f75389ed6147a0 2023-04-25 02:55:11,931 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/C/1616975a2e1b498b82f75389ed6147a0, entries=150, sequenceid=146, filesize=11.8 K 2023-04-25 02:55:11,932 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~360.33 KB/368976, currentSize=60.38 KB/61830 for 9d5b411b5fe67e5bb007f4d36e3e11e5 in 466ms, sequenceid=146, compaction requested=true 2023-04-25 02:55:11,932 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9d5b411b5fe67e5bb007f4d36e3e11e5: 2023-04-25 02:55:11,932 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. 2023-04-25 02:55:11,932 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:55:11,932 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:55:11,932 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:11,934 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:55:11,934 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:55:11,935 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:11,935 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:11,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:11,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:11,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:11,936 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:11,936 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:55:11,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:11,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:11,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:11,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:11,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:11,937 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:55:11,937 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@799887e2[Count = 0] remaining members to acquire global barrier 2023-04-25 02:55:11,937 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:55:11,937 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:11,939 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:11,939 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:11,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:11,939 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:55:11,939 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:55:11,939 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:11,939 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:55:11,939 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:55:11,940 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:11,940 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:55:11,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:11,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:11,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:11,940 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:11,940 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:55:11,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:11,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:11,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:11,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:11,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:11,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:11,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:11,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:55:11,942 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:55:11,942 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:55:11,943 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:55:11,943 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:55:11,943 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:55:11,943 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:55:11,944 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:11,944 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:11,944 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:11,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:11,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:11,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:11,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:11,944 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:11,945 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:11,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:11,945 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:11,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:11,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:11,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:11,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:11,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:11,946 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:11,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:11,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:11,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:11,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:11,948 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:11,948 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:11,948 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:11,948 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:11,948 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:11,948 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:55:11,948 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:55:11,948 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:11,948 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:11,948 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:55:11,948 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:11,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:11,948 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:11,948 DEBUG [Thread-2260] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:55:11,948 DEBUG [Thread-2260] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:55:11,948 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:11,948 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:11,949 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:11,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:12,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(9158): Flush requested on 9d5b411b5fe67e5bb007f4d36e3e11e5 2023-04-25 02:55:12,019 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 9d5b411b5fe67e5bb007f4d36e3e11e5 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-25 02:55:12,019 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9d5b411b5fe67e5bb007f4d36e3e11e5, store=A 2023-04-25 02:55:12,020 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:55:12,020 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9d5b411b5fe67e5bb007f4d36e3e11e5, store=B 2023-04-25 02:55:12,020 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:55:12,020 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9d5b411b5fe67e5bb007f4d36e3e11e5, store=C 2023-04-25 02:55:12,020 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:55:12,049 DEBUG [Thread-2260] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:55:12,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:55:12,050 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:55:12,051 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:55:12,051 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:55:12,051 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:55:12,051 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:12,051 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:55:12,051 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:55:12,052 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:12,052 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:12,054 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:12,055 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:12,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:12,055 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:12,055 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:12,055 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:55:12,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:12,055 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:12,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:55:12,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:12,056 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:55:12,056 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:55:12,056 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:55:12,056 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:55:12,056 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:12,056 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:55:12,056 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:55:12,056 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:55:12,056 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. 2023-04-25 02:55:12,056 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. started... 2023-04-25 02:55:12,057 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. as already flushing 2023-04-25 02:55:12,057 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. 2023-04-25 02:55:12,057 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:55:12,057 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:55:12,057 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:12,057 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:55:12,057 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:55:12,058 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:12,058 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:12,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:12,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:12,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:12,058 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:12,058 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:55:12,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:12,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:12,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:12,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:12,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:12,060 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:55:12,060 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@646041f3[Count = 0] remaining members to acquire global barrier 2023-04-25 02:55:12,060 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:55:12,060 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:12,061 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:12,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54886 deadline: 1682391372054, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:12,061 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:12,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54940 deadline: 1682391372056, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:12,062 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:12,062 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:12,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:12,062 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:55:12,062 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:12,062 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:55:12,062 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:55:12,062 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:55:12,063 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:12,063 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:55:12,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:12,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:12,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:12,063 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:12,063 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:55:12,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:12,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:12,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:12,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:12,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:12,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:12,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:12,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:55:12,065 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:55:12,065 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:55:12,065 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:55:12,065 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:55:12,065 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:55:12,065 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:55:12,067 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:12,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54894 deadline: 1682391372057, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:12,071 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:12,071 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:12,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:12,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:12,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:12,071 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:12,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:12,071 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:12,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:12,072 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:12,072 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:12,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:12,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:12,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:12,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:12,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:12,074 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:12,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:12,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:12,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:12,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:12,080 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:12,080 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:12,080 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:12,080 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:12,080 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:12,080 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:55:12,080 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:55:12,080 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:12,080 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:12,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:12,080 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:12,080 DEBUG [Thread-2260] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:55:12,080 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:12,081 DEBUG [Thread-2260] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:55:12,081 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:12,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:12,081 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:12,081 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:12,165 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:12,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54886 deadline: 1682391372162, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:12,165 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:12,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54940 deadline: 1682391372162, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:12,171 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:12,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54894 deadline: 1682391372168, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:12,181 DEBUG [Thread-2260] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:55:12,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:55:12,183 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:55:12,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:55:12,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:55:12,184 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:55:12,184 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:12,184 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:55:12,184 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:55:12,184 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:12,185 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:12,186 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:12,186 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:12,186 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:12,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:12,186 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:12,186 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:55:12,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:12,186 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:12,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:55:12,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:12,187 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:55:12,187 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:55:12,187 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:55:12,187 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:55:12,187 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:12,187 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:55:12,187 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:55:12,187 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:55:12,187 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. 2023-04-25 02:55:12,187 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. started... 2023-04-25 02:55:12,187 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. as already flushing 2023-04-25 02:55:12,187 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. 2023-04-25 02:55:12,187 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:55:12,187 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:55:12,187 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:12,188 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:55:12,188 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:55:12,189 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:12,189 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:12,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:12,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:12,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:12,189 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:12,189 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:55:12,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:12,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:12,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:12,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:12,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:12,191 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:55:12,191 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4af31eca[Count = 0] remaining members to acquire global barrier 2023-04-25 02:55:12,191 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:55:12,191 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:12,193 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:12,193 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:12,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:12,193 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:55:12,193 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:55:12,193 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:12,193 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:55:12,193 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:55:12,195 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:55:12,195 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:12,195 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:12,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:12,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:12,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:12,195 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:55:12,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:12,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:12,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:12,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:12,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:12,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:12,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:12,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:55:12,199 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:55:12,199 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:55:12,199 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:55:12,199 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:55:12,199 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:55:12,199 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:55:12,200 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:12,200 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:12,200 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:12,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:12,200 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:12,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:12,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:12,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:12,201 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:12,201 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:12,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:12,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:12,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:12,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:12,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:12,202 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:12,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:12,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:12,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:12,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:12,205 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:12,205 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:12,205 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:12,205 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:12,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:12,205 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:12,205 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:12,205 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:55:12,205 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:55:12,205 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:12,205 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:12,205 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:12,205 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:12,205 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:12,205 DEBUG [Thread-2260] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:55:12,205 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:55:12,205 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:12,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:12,205 DEBUG [Thread-2260] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:55:12,306 DEBUG [Thread-2260] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:55:12,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:55:12,308 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:55:12,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:55:12,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:55:12,309 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:55:12,309 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:12,309 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:55:12,309 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:55:12,310 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:12,310 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:12,312 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:12,312 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:12,312 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:12,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:12,312 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:12,312 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:55:12,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:12,312 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:12,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:55:12,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:12,313 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:55:12,313 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:55:12,313 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:55:12,313 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:55:12,313 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:12,313 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:55:12,313 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:55:12,313 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:55:12,313 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. 2023-04-25 02:55:12,313 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. started... 2023-04-25 02:55:12,314 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. as already flushing 2023-04-25 02:55:12,314 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. 2023-04-25 02:55:12,314 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:55:12,314 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:55:12,314 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:12,314 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:55:12,314 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:55:12,315 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:12,315 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:12,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:12,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:12,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:12,315 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:12,315 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:55:12,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:12,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:12,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:12,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:12,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:12,317 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:55:12,317 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1c889102[Count = 0] remaining members to acquire global barrier 2023-04-25 02:55:12,317 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:55:12,317 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:12,318 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:12,318 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:12,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:12,318 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:55:12,318 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:12,318 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:55:12,318 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:55:12,318 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:55:12,319 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:12,319 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:55:12,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:12,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:12,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:12,319 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:12,320 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:55:12,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:12,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:12,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:12,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:12,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:12,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:12,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:12,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:55:12,323 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:55:12,323 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:55:12,323 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:55:12,323 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:55:12,323 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:55:12,323 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:55:12,326 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:12,326 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:12,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:12,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:12,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:12,326 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:12,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:12,326 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:12,326 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:12,326 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:12,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:12,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:12,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:12,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:12,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:12,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:12,327 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:12,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:12,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:12,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:12,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:12,329 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:12,329 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:12,329 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:12,329 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:12,329 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:12,330 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:12,330 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:12,330 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:55:12,330 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:12,330 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:55:12,330 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:55:12,330 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:12,330 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:12,330 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:12,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:12,330 DEBUG [Thread-2260] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:55:12,330 DEBUG [Thread-2260] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:55:12,330 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:12,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:12,368 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:12,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54886 deadline: 1682391372366, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:12,368 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:12,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54940 deadline: 1682391372366, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:12,375 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:12,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54894 deadline: 1682391372372, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:12,430 DEBUG [Thread-2260] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:55:12,431 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:55:12,431 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304258ba9b8c9f4574eee9b96607ef8a8e6b6 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304258ba9b8c9f4574eee9b96607ef8a8e6b6 2023-04-25 02:55:12,433 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:55:12,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:55:12,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:55:12,434 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:55:12,434 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:12,434 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:55:12,434 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:55:12,434 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:12,434 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:12,435 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:12,435 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:12,435 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:12,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:12,435 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:12,435 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:55:12,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:12,436 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:12,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:55:12,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:12,437 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:55:12,437 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:55:12,437 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:55:12,437 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:55:12,437 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:12,437 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:55:12,437 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:55:12,437 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:55:12,437 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. 2023-04-25 02:55:12,437 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. started... 2023-04-25 02:55:12,437 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. as already flushing 2023-04-25 02:55:12,438 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. 2023-04-25 02:55:12,438 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:55:12,438 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:55:12,438 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:12,438 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:55:12,438 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:55:12,439 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:12,439 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:12,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:12,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:12,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:12,439 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:12,439 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:55:12,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:12,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:12,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:12,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:12,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:12,441 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:55:12,441 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2112a2e6[Count = 0] remaining members to acquire global barrier 2023-04-25 02:55:12,441 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:55:12,441 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:12,443 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:12,443 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:12,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:12,443 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:55:12,443 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:55:12,443 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:55:12,443 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:12,444 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:55:12,445 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:12,445 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:55:12,445 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:12,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:12,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:12,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:12,445 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:55:12,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:12,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:12,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:12,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:12,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:12,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:12,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:12,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:55:12,447 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:55:12,447 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:55:12,447 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:55:12,447 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:55:12,447 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:55:12,447 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:55:12,449 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:12,449 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:12,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:12,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:12,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:12,449 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:12,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:12,449 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:12,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:12,449 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:12,449 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:12,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:12,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:12,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:12,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:12,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:12,450 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:12,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:12,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:12,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:12,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:12,456 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:12,456 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:12,456 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:12,456 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:12,456 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:12,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:12,456 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:12,456 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:12,456 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:55:12,457 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:55:12,457 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:12,457 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:12,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:12,457 DEBUG [Thread-2260] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:55:12,457 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:55:12,457 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:12,457 DEBUG [Thread-2260] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:55:12,457 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:12,457 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:12,557 DEBUG [Thread-2260] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:55:12,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:55:12,559 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:55:12,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:55:12,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:55:12,560 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:55:12,560 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:12,560 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:55:12,561 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:55:12,561 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:12,561 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:12,562 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:12,562 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:12,562 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:12,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:12,562 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:12,562 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:55:12,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:12,563 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:12,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:55:12,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:12,563 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:55:12,563 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:55:12,563 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:55:12,563 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:55:12,563 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:12,564 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:55:12,564 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:55:12,564 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:55:12,564 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. 2023-04-25 02:55:12,564 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. started... 2023-04-25 02:55:12,564 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. as already flushing 2023-04-25 02:55:12,564 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. 2023-04-25 02:55:12,564 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:55:12,564 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:55:12,564 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:12,564 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:55:12,564 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:55:12,565 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:12,565 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:12,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:12,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:12,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:12,566 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:12,566 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:55:12,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:12,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:12,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:12,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:12,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:12,567 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:55:12,567 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:55:12,567 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@471b10ee[Count = 0] remaining members to acquire global barrier 2023-04-25 02:55:12,567 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:12,570 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:12,570 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:12,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:12,570 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:55:12,570 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:55:12,570 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:55:12,570 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:12,570 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:55:12,571 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:12,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:12,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:12,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:12,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:12,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:12,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:12,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:12,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:12,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:12,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:12,573 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:55:12,573 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:12,573 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:55:12,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:55:12,574 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:55:12,574 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:55:12,574 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:55:12,574 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:55:12,574 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:55:12,574 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:55:12,576 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:12,576 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:12,576 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:12,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:12,576 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:12,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:12,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:12,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:12,576 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:12,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:12,576 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:12,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:12,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:12,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:12,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:12,577 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:12,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:12,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:12,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:12,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:12,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:12,579 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:12,579 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:12,579 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:12,579 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:12,579 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:12,579 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:12,579 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:55:12,579 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:55:12,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:12,579 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:12,580 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:12,580 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:12,580 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:12,580 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:12,580 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:12,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:12,580 DEBUG [Thread-2260] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:55:12,580 DEBUG [Thread-2260] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:55:12,580 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:55:12,673 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:12,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54886 deadline: 1682391372670, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:12,674 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:12,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54940 deadline: 1682391372670, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:12,680 DEBUG [Thread-2260] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:55:12,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:55:12,681 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:12,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54894 deadline: 1682391372678, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:12,682 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:55:12,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:55:12,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:55:12,683 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:55:12,683 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:12,683 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:55:12,683 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:55:12,684 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:12,684 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:12,686 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:12,686 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:12,686 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:12,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:12,686 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:12,686 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:55:12,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:12,687 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:12,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:55:12,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:12,687 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:55:12,687 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:55:12,687 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:55:12,687 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:55:12,687 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:12,687 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:55:12,687 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:55:12,687 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:55:12,688 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. 2023-04-25 02:55:12,688 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. started... 2023-04-25 02:55:12,688 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. as already flushing 2023-04-25 02:55:12,688 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. 2023-04-25 02:55:12,688 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:55:12,688 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:55:12,688 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:12,688 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:55:12,688 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:55:12,689 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:12,689 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:12,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:12,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:12,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:12,690 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:12,690 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:55:12,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:12,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:12,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:12,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:12,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:12,691 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:55:12,691 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@154c5389[Count = 0] remaining members to acquire global barrier 2023-04-25 02:55:12,691 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:55:12,691 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:12,692 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:12,692 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:12,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:12,692 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:55:12,692 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:55:12,692 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:55:12,692 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:12,692 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:55:12,693 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:12,693 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:55:12,693 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:12,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:12,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:12,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:12,693 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:55:12,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:12,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:12,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:12,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:12,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:12,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:12,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:12,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:55:12,696 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:55:12,696 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:55:12,696 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:55:12,696 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:55:12,696 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:55:12,696 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:55:12,698 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:12,698 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:12,698 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:12,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:12,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:12,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:12,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:12,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:12,699 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:12,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:12,699 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:12,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:12,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:12,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:12,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:12,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:12,700 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:12,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:12,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:12,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:12,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:12,702 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:12,702 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:12,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:12,708 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:12,708 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:12,708 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:12,708 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:12,708 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:12,708 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:12,708 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:55:12,708 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:55:12,708 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:12,708 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:12,708 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:12,708 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:12,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:12,708 DEBUG [Thread-2260] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:55:12,709 DEBUG [Thread-2260] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:55:12,712 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:55:12,809 DEBUG [Thread-2260] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:55:12,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:55:12,811 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:55:12,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:55:12,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:55:12,812 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:55:12,812 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:12,812 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:55:12,812 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:55:12,812 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:12,812 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:12,814 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:12,814 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:12,814 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:12,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:12,814 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:12,814 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:55:12,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:12,815 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:12,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:55:12,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:12,815 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:55:12,815 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:55:12,815 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:55:12,815 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:55:12,815 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:12,816 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:55:12,816 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:55:12,816 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:55:12,816 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. 2023-04-25 02:55:12,816 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. started... 2023-04-25 02:55:12,816 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. as already flushing 2023-04-25 02:55:12,816 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. 2023-04-25 02:55:12,816 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:55:12,816 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:55:12,816 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:12,816 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:55:12,816 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:55:12,817 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:12,817 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:12,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:12,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:12,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:12,818 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:12,818 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:55:12,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:12,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:12,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:12,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:12,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:12,819 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:55:12,819 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4d14d0a3[Count = 0] remaining members to acquire global barrier 2023-04-25 02:55:12,819 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:55:12,819 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:12,822 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:12,822 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:12,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:12,822 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:55:12,822 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:55:12,822 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:55:12,822 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:12,822 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:55:12,823 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:12,823 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:55:12,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:12,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:12,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:12,823 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:12,823 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:55:12,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:12,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:12,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:12,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:12,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:12,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:12,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:12,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:55:12,825 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:55:12,825 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:55:12,825 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:55:12,825 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:55:12,825 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:55:12,825 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:55:12,827 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:12,827 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:12,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:12,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:12,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:12,827 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:12,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:12,827 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:12,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:12,827 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:12,827 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:12,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:12,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:12,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:12,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:12,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:12,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:12,828 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:12,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:12,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:12,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:12,831 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:12,831 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:12,831 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:12,831 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:12,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:12,831 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:55:12,831 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:55:12,831 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:12,831 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:12,831 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:12,831 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:12,831 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:12,831 DEBUG [Thread-2260] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:55:12,831 DEBUG [Thread-2260] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:55:12,831 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:12,831 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:12,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:12,831 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:12,837 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=160, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/.tmp/A/aa223b4266764748a286c96dc8f5f842 2023-04-25 02:55:12,847 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=160 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/.tmp/B/3c2de4efd527437f9f4188dc83ab23f8 2023-04-25 02:55:12,931 DEBUG [Thread-2260] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:55:12,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:55:12,933 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:55:12,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:55:12,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:55:12,936 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:55:12,936 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:12,936 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:55:12,936 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:55:12,936 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:12,936 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:12,945 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:12,945 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:12,945 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:12,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:12,945 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:12,945 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:55:12,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:12,946 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:12,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:55:12,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:12,946 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:55:12,946 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:55:12,946 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:55:12,946 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:55:12,946 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:12,946 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:55:12,947 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:55:12,947 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:55:12,947 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. 2023-04-25 02:55:12,947 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. started... 2023-04-25 02:55:12,947 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. as already flushing 2023-04-25 02:55:12,947 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. 2023-04-25 02:55:12,947 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:55:12,947 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:55:12,947 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:12,947 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:55:12,947 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:55:12,948 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:12,949 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:12,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:12,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:12,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:12,949 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:12,949 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:55:12,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:12,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:12,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:12,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:12,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:12,950 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:55:12,950 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@79a54e75[Count = 0] remaining members to acquire global barrier 2023-04-25 02:55:12,950 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:55:12,950 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:12,951 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:12,952 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:12,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:12,952 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:55:12,952 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:55:12,952 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:55:12,953 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:12,953 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:55:12,955 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:12,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:12,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:12,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:12,955 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:55:12,955 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:12,955 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:55:12,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:12,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:12,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:12,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:12,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:12,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:12,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:12,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:55:12,958 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:55:12,958 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:55:12,958 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:55:12,958 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:55:12,958 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:55:12,958 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:55:12,959 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:12,959 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:12,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:12,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:12,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:12,959 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:12,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:12,959 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:12,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:12,960 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:12,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:12,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:12,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:12,961 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:12,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:12,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:12,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:12,962 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:12,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:12,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:12,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:12,965 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:12,965 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:12,965 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:12,965 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:55:12,965 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:55:12,965 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:12,965 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:12,965 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:12,966 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:12,966 DEBUG [Thread-2260] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:55:12,966 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:12,965 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:12,965 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:12,965 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:12,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:12,966 DEBUG [Thread-2260] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:55:12,966 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:12,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:13,066 DEBUG [Thread-2260] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:55:13,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:55:13,068 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:55:13,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:55:13,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:55:13,069 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:55:13,069 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:13,069 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:55:13,069 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:55:13,069 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:13,069 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:13,070 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:13,070 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:13,070 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:13,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:13,071 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:13,071 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:55:13,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:13,071 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:13,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:55:13,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:13,072 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:55:13,072 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:55:13,072 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:55:13,072 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:55:13,072 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:13,072 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:55:13,072 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:55:13,072 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:55:13,072 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. 2023-04-25 02:55:13,072 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. started... 2023-04-25 02:55:13,072 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. as already flushing 2023-04-25 02:55:13,072 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. 2023-04-25 02:55:13,072 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:55:13,073 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:55:13,073 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:13,073 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:55:13,073 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:55:13,075 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:13,075 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:13,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:13,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:13,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:13,076 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:13,076 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:55:13,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:13,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:13,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:13,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:13,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:13,077 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:55:13,077 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@66a6adcf[Count = 0] remaining members to acquire global barrier 2023-04-25 02:55:13,077 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:55:13,077 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:13,078 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:13,078 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:13,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:13,078 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:55:13,078 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:55:13,078 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:13,078 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:55:13,078 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:55:13,079 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:13,079 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:55:13,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:13,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:13,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:13,079 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:13,080 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:55:13,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:13,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:13,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:13,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:13,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:13,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:13,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:13,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:55:13,081 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:55:13,081 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:55:13,081 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:55:13,081 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:55:13,081 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:55:13,082 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:55:13,083 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:13,083 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:13,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:13,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:13,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:13,083 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:13,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:13,083 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:13,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:13,083 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:13,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:13,083 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:13,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:13,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:13,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:13,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:13,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:13,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:13,085 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:13,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:13,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:13,088 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:13,088 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:13,088 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:13,088 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:13,088 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:13,088 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:13,088 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:55:13,088 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:55:13,088 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:13,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:13,088 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:13,088 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:13,088 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:13,088 DEBUG [Thread-2260] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:55:13,088 DEBUG [Thread-2260] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:55:13,088 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:13,088 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:13,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:13,181 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:13,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54940 deadline: 1682391373178, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:13,181 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:13,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54886 deadline: 1682391373178, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:13,188 DEBUG [Thread-2260] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:55:13,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:55:13,190 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:55:13,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:55:13,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:55:13,191 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:55:13,191 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:13,191 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:55:13,191 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:55:13,192 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:13,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54894 deadline: 1682391373186, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:13,192 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:13,192 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:13,193 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:13,193 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:13,193 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:13,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:13,193 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:13,193 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:55:13,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:13,194 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:13,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:55:13,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:13,194 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:55:13,194 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:55:13,194 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:55:13,194 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:55:13,195 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:13,195 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:55:13,195 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:55:13,195 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:55:13,195 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. 2023-04-25 02:55:13,195 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. started... 2023-04-25 02:55:13,195 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. as already flushing 2023-04-25 02:55:13,195 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. 2023-04-25 02:55:13,195 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:55:13,195 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:55:13,195 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:13,195 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:55:13,195 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:55:13,196 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:13,196 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:13,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:13,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:13,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:13,197 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:13,197 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:55:13,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:13,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:13,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:13,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:13,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:13,198 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:55:13,198 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@11016a89[Count = 0] remaining members to acquire global barrier 2023-04-25 02:55:13,198 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:55:13,198 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:13,199 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:13,200 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:13,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:13,200 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:55:13,200 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:55:13,200 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:55:13,200 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:13,200 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:55:13,203 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:13,203 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:55:13,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:13,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:13,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:13,203 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:13,204 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:55:13,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:13,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:13,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:13,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:13,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:13,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:13,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:13,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:55:13,206 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:55:13,206 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:55:13,206 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:55:13,206 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:55:13,206 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:55:13,206 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:55:13,207 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:13,207 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:13,207 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:13,207 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:13,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:13,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:13,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:13,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:13,208 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:13,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:13,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:13,208 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:13,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:13,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:13,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:13,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:13,209 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:13,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:13,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:13,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:13,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:13,212 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:13,212 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:13,213 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:13,212 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:13,213 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:13,213 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:55:13,213 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:13,213 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:13,213 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:55:13,213 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:13,213 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:55:13,213 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:13,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:13,213 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:13,213 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:13,213 DEBUG [Thread-2260] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:55:13,213 DEBUG [Thread-2260] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:55:13,214 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:13,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:13,257 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=160 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/.tmp/C/973a1219537a4e54b56706439e8a7866 2023-04-25 02:55:13,261 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/.tmp/A/aa223b4266764748a286c96dc8f5f842 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/A/aa223b4266764748a286c96dc8f5f842 2023-04-25 02:55:13,265 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/A/aa223b4266764748a286c96dc8f5f842, entries=200, sequenceid=160, filesize=32.1 K 2023-04-25 02:55:13,266 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/.tmp/B/3c2de4efd527437f9f4188dc83ab23f8 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/B/3c2de4efd527437f9f4188dc83ab23f8 2023-04-25 02:55:13,269 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/B/3c2de4efd527437f9f4188dc83ab23f8, entries=150, sequenceid=160, filesize=11.8 K 2023-04-25 02:55:13,269 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/.tmp/C/973a1219537a4e54b56706439e8a7866 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/C/973a1219537a4e54b56706439e8a7866 2023-04-25 02:55:13,273 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/C/973a1219537a4e54b56706439e8a7866, entries=150, sequenceid=160, filesize=11.8 K 2023-04-25 02:55:13,274 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 9d5b411b5fe67e5bb007f4d36e3e11e5 in 1255ms, sequenceid=160, compaction requested=true 2023-04-25 02:55:13,274 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 9d5b411b5fe67e5bb007f4d36e3e11e5: 2023-04-25 02:55:13,274 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 02:55:13,274 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-25 02:55:13,274 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-25 02:55:13,274 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 02:55:13,274 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 02:55:13,275 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 110412 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-25 02:55:13,275 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47982 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-25 02:55:13,275 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HStore(1912): 9d5b411b5fe67e5bb007f4d36e3e11e5/A is initiating minor compaction (all files) 2023-04-25 02:55:13,275 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HStore(1912): 9d5b411b5fe67e5bb007f4d36e3e11e5/B is initiating minor compaction (all files) 2023-04-25 02:55:13,275 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 9d5b411b5fe67e5bb007f4d36e3e11e5/A in TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. 2023-04-25 02:55:13,275 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 9d5b411b5fe67e5bb007f4d36e3e11e5/B in TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. 2023-04-25 02:55:13,275 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/A/fde37d724a4447e189b46f2fe3ded178, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/A/0819a75a91174ab38d49623fbe728436, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/A/b82f6c1510c545ba8f5be887bb396bbe, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/A/aa223b4266764748a286c96dc8f5f842] into tmpdir=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/.tmp, totalSize=107.8 K 2023-04-25 02:55:13,275 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/B/4452115a19bd4f5e83148150b3a27076, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/B/886195b72dee4da182fa19a6b333d181, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/B/765c196572fd4dc9999fca242f93b77b, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/B/3c2de4efd527437f9f4188dc83ab23f8] into tmpdir=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/.tmp, totalSize=46.9 K 2023-04-25 02:55:13,276 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting fde37d724a4447e189b46f2fe3ded178, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=98, earliestPutTs=1682391308306 2023-04-25 02:55:13,276 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting 4452115a19bd4f5e83148150b3a27076, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=98, earliestPutTs=1682391308306 2023-04-25 02:55:13,276 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting 0819a75a91174ab38d49623fbe728436, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=109, earliestPutTs=1682391309168 2023-04-25 02:55:13,276 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting 886195b72dee4da182fa19a6b333d181, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=109, earliestPutTs=1682391309168 2023-04-25 02:55:13,277 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting b82f6c1510c545ba8f5be887bb396bbe, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=146, earliestPutTs=1682391310550 2023-04-25 02:55:13,277 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting 765c196572fd4dc9999fca242f93b77b, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=146, earliestPutTs=1682391310550 2023-04-25 02:55:13,277 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting aa223b4266764748a286c96dc8f5f842, keycount=200, bloomtype=ROW, size=32.1 K, encoding=NONE, compression=NONE, seqNum=160, earliestPutTs=1682391311699 2023-04-25 02:55:13,277 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting 3c2de4efd527437f9f4188dc83ab23f8, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=160, earliestPutTs=1682391311699 2023-04-25 02:55:13,297 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] throttle.PressureAwareThroughputController(145): 9d5b411b5fe67e5bb007f4d36e3e11e5#B#compaction#319 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 53.85 MB/second 2023-04-25 02:55:13,313 DEBUG [Thread-2260] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:55:13,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:55:13,315 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:55:13,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:55:13,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:55:13,317 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:55:13,317 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:13,317 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:55:13,317 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:55:13,319 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] throttle.PressureAwareThroughputController(145): 9d5b411b5fe67e5bb007f4d36e3e11e5#A#compaction#318 average throughput is 0.79 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-25 02:55:13,319 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:13,319 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:13,322 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:13,322 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:13,323 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:13,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:13,325 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:13,325 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:55:13,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:13,326 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:13,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:55:13,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:13,327 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:55:13,328 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:55:13,328 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:55:13,328 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:55:13,328 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:13,328 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:55:13,329 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/.tmp/B/3703b5da4c594ed68d58845cfd8cb0ad as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/B/3703b5da4c594ed68d58845cfd8cb0ad 2023-04-25 02:55:13,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(9158): Flush requested on 9d5b411b5fe67e5bb007f4d36e3e11e5 2023-04-25 02:55:13,330 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 9d5b411b5fe67e5bb007f4d36e3e11e5 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-25 02:55:13,330 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9d5b411b5fe67e5bb007f4d36e3e11e5, store=A 2023-04-25 02:55:13,330 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:55:13,330 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9d5b411b5fe67e5bb007f4d36e3e11e5, store=B 2023-04-25 02:55:13,330 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:55:13,330 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9d5b411b5fe67e5bb007f4d36e3e11e5, store=C 2023-04-25 02:55:13,330 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:55:13,336 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:55:13,336 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:55:13,336 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. 2023-04-25 02:55:13,336 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. started... 2023-04-25 02:55:13,336 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. as already flushing 2023-04-25 02:55:13,336 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. 2023-04-25 02:55:13,336 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:55:13,336 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:55:13,336 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:13,336 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:55:13,336 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:55:13,341 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 9d5b411b5fe67e5bb007f4d36e3e11e5/B of 9d5b411b5fe67e5bb007f4d36e3e11e5 into 3703b5da4c594ed68d58845cfd8cb0ad(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 02:55:13,341 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9d5b411b5fe67e5bb007f4d36e3e11e5: 2023-04-25 02:55:13,341 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5., storeName=9d5b411b5fe67e5bb007f4d36e3e11e5/B, priority=12, startTime=1682391313274; duration=0sec 2023-04-25 02:55:13,341 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 02:55:13,341 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-25 02:55:13,341 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:13,344 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:13,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:13,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:13,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:13,344 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:13,344 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:55:13,344 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47982 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-25 02:55:13,344 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HStore(1912): 9d5b411b5fe67e5bb007f4d36e3e11e5/C is initiating minor compaction (all files) 2023-04-25 02:55:13,344 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 9d5b411b5fe67e5bb007f4d36e3e11e5/C in TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. 2023-04-25 02:55:13,344 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/C/73446cf519cf44218916175b681de98f, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/C/68d78dd5c2834816b8ca1a20c1cbe8b4, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/C/1616975a2e1b498b82f75389ed6147a0, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/C/973a1219537a4e54b56706439e8a7866] into tmpdir=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/.tmp, totalSize=46.9 K 2023-04-25 02:55:13,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:13,345 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting 73446cf519cf44218916175b681de98f, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=98, earliestPutTs=1682391308306 2023-04-25 02:55:13,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:13,345 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting 68d78dd5c2834816b8ca1a20c1cbe8b4, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=109, earliestPutTs=1682391309168 2023-04-25 02:55:13,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:13,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:13,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:13,346 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:55:13,346 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7a7b31c2[Count = 0] remaining members to acquire global barrier 2023-04-25 02:55:13,346 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:55:13,346 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:13,346 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting 1616975a2e1b498b82f75389ed6147a0, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=146, earliestPutTs=1682391310550 2023-04-25 02:55:13,348 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:13,348 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:13,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:13,348 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:55:13,348 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:55:13,348 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:13,348 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:55:13,348 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:55:13,350 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:13,350 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:55:13,350 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:13,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:13,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:13,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:13,350 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:55:13,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:13,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:13,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:13,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:13,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:13,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:13,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:13,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:55:13,352 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:55:13,352 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:55:13,352 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:55:13,352 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:55:13,352 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting 973a1219537a4e54b56706439e8a7866, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=160, earliestPutTs=1682391311699 2023-04-25 02:55:13,352 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:55:13,352 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:55:13,358 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:13,358 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:13,358 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:13,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:13,358 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:13,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:13,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:13,360 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023042523ba0fe2ca7448899fe9db2e7457bffa to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042523ba0fe2ca7448899fe9db2e7457bffa 2023-04-25 02:55:13,364 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:13,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:13,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:13,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:13,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:13,365 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:13,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:13,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:13,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:13,366 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] throttle.PressureAwareThroughputController(145): 9d5b411b5fe67e5bb007f4d36e3e11e5#C#compaction#321 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-25 02:55:13,366 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:13,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:13,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:13,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:13,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:13,369 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:13,369 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:13,369 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:13,369 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:13,369 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:13,369 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:13,369 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:13,369 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:55:13,369 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:55:13,369 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:13,369 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:13,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:13,369 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:13,369 DEBUG [Thread-2260] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:55:13,369 DEBUG [Thread-2260] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:55:13,369 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:13,369 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:13,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:13,382 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=180, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/.tmp/A/a15c8b364906459f9302cbcc776860f8 2023-04-25 02:55:13,388 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/.tmp/C/cc1f84c3c938484f9a6bede5da083ae5 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/C/cc1f84c3c938484f9a6bede5da083ae5 2023-04-25 02:55:13,393 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 9d5b411b5fe67e5bb007f4d36e3e11e5/C of 9d5b411b5fe67e5bb007f4d36e3e11e5 into cc1f84c3c938484f9a6bede5da083ae5(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 02:55:13,393 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9d5b411b5fe67e5bb007f4d36e3e11e5: 2023-04-25 02:55:13,393 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5., storeName=9d5b411b5fe67e5bb007f4d36e3e11e5/C, priority=12, startTime=1682391313274; duration=0sec 2023-04-25 02:55:13,393 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 02:55:13,399 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:13,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54884 deadline: 1682391373390, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:13,402 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:13,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54924 deadline: 1682391373391, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:13,469 DEBUG [Thread-2260] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:55:13,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:55:13,472 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:55:13,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:55:13,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:55:13,473 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:55:13,473 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:13,473 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:55:13,473 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:55:13,474 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:13,474 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:13,475 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:13,475 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:13,475 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:13,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:13,475 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:13,475 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:55:13,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:13,476 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:13,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:55:13,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:13,476 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:55:13,476 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:55:13,476 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:55:13,476 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:55:13,476 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:13,477 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:55:13,477 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:55:13,477 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:55:13,477 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. 2023-04-25 02:55:13,477 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. started... 2023-04-25 02:55:13,477 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. as already flushing 2023-04-25 02:55:13,477 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. 2023-04-25 02:55:13,477 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:55:13,477 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:55:13,477 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:13,477 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:55:13,477 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:55:13,478 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:13,478 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:13,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:13,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:13,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:13,479 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:13,479 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:55:13,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:13,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:13,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:13,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:13,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:13,482 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:55:13,482 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@297e1e1e[Count = 0] remaining members to acquire global barrier 2023-04-25 02:55:13,482 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:55:13,482 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:13,484 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:13,484 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:13,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:13,484 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:55:13,484 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:55:13,484 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:13,484 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:55:13,484 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:55:13,486 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:13,486 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:55:13,486 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:13,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:13,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:13,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:13,486 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:55:13,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:13,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:13,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:13,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:13,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:13,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:13,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:13,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:55:13,490 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:55:13,490 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:55:13,490 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:55:13,490 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:55:13,490 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:55:13,490 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:55:13,491 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:13,491 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:13,491 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:13,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:13,491 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:13,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:13,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:13,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:13,492 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:13,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:13,492 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:13,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:13,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:13,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:13,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:13,493 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:13,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:13,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:13,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:13,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:13,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:13,495 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:13,495 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:13,495 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:13,495 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:13,495 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:13,495 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:13,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:13,495 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:13,495 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:55:13,496 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:55:13,495 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:13,496 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:13,496 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:13,496 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:13,496 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:13,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:13,496 DEBUG [Thread-2260] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:55:13,496 DEBUG [Thread-2260] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:55:13,496 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:55:13,500 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:13,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54884 deadline: 1682391373500, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:13,506 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:13,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54924 deadline: 1682391373503, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:13,596 DEBUG [Thread-2260] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:55:13,597 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:55:13,598 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:55:13,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:55:13,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:55:13,600 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:55:13,600 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:13,600 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:55:13,600 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:55:13,600 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:13,600 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:13,602 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:13,602 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:13,602 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:13,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:13,603 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:13,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:13,603 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:55:13,603 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:13,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:55:13,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:13,604 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:55:13,604 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:55:13,604 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:55:13,604 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:55:13,604 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:13,604 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:55:13,604 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:55:13,604 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:55:13,604 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. 2023-04-25 02:55:13,604 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. started... 2023-04-25 02:55:13,604 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. as already flushing 2023-04-25 02:55:13,605 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. 2023-04-25 02:55:13,605 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:55:13,605 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:55:13,605 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:13,605 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:55:13,605 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:55:13,606 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:13,606 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:13,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:13,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:13,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:13,606 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:13,606 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:55:13,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:13,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:13,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:13,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:13,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:13,608 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:55:13,608 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7ae5062b[Count = 0] remaining members to acquire global barrier 2023-04-25 02:55:13,608 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:55:13,608 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:13,609 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:13,609 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:13,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:13,609 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:55:13,609 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:55:13,609 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:55:13,609 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:13,609 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:55:13,610 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:13,610 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:55:13,610 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:13,610 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:55:13,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:13,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:13,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:13,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:13,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:13,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:13,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:13,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:13,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:13,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:13,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:55:13,613 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:55:13,613 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:55:13,613 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:55:13,613 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:55:13,613 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:55:13,613 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:55:13,615 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:13,615 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:13,615 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:13,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:13,615 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:13,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:13,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:13,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:13,616 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:13,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:13,616 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:13,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:13,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:13,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:13,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:13,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:13,618 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:13,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:13,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:13,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:13,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:13,620 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:13,620 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:13,620 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:13,620 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:13,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:13,620 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:13,620 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:13,620 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:55:13,620 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:55:13,620 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:13,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:13,620 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:13,621 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:13,621 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:13,621 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:13,621 DEBUG [Thread-2260] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:55:13,621 DEBUG [Thread-2260] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:55:13,621 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:55:13,621 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:13,705 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:13,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54884 deadline: 1682391373702, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:13,711 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:13,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54924 deadline: 1682391373708, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:13,721 DEBUG [Thread-2260] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:55:13,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:55:13,723 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:55:13,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:55:13,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:55:13,724 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:55:13,724 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:13,724 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:55:13,724 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:55:13,724 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:13,724 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:13,726 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:13,726 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:13,726 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:13,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:13,726 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:13,726 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:55:13,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:13,726 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:13,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:55:13,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:13,727 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:55:13,727 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:55:13,727 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:55:13,727 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:55:13,727 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:13,727 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:55:13,727 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:55:13,727 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:55:13,727 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. 2023-04-25 02:55:13,727 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. started... 2023-04-25 02:55:13,728 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. as already flushing 2023-04-25 02:55:13,728 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. 2023-04-25 02:55:13,728 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:55:13,728 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:55:13,728 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:13,728 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:55:13,728 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:55:13,729 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:13,729 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:13,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:13,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:13,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:13,729 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:13,730 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:55:13,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:13,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:13,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:13,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:13,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:13,731 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:55:13,731 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@60aa7edc[Count = 0] remaining members to acquire global barrier 2023-04-25 02:55:13,731 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:55:13,731 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:13,733 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:13,733 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:13,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:13,733 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:55:13,733 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:55:13,733 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:55:13,733 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:13,733 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:55:13,736 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:13,736 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:55:13,736 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:13,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:13,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:13,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:13,736 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:55:13,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:13,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:13,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:13,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:13,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:13,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:13,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:13,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:55:13,739 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:55:13,739 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:55:13,739 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:55:13,739 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:55:13,739 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:55:13,739 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:55:13,740 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:13,740 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:13,740 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:13,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:13,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:13,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:13,740 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:13,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:13,741 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:13,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:13,741 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:13,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:13,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:13,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:13,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:13,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:13,742 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:13,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:13,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:13,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:13,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:13,744 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:13,744 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:13,744 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:13,744 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:13,744 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:13,744 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:13,744 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:55:13,744 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:55:13,744 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:55:13,744 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:13,744 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:13,745 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:13,745 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:13,745 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:13,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:13,745 DEBUG [Thread-2260] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:55:13,745 DEBUG [Thread-2260] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:55:13,745 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:13,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:13,770 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/.tmp/A/82ca2bf000e646d9b901407e5783065f as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/A/82ca2bf000e646d9b901407e5783065f 2023-04-25 02:55:13,774 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 9d5b411b5fe67e5bb007f4d36e3e11e5/A of 9d5b411b5fe67e5bb007f4d36e3e11e5 into 82ca2bf000e646d9b901407e5783065f(size=25.6 K), total size for store is 25.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 02:55:13,774 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9d5b411b5fe67e5bb007f4d36e3e11e5: 2023-04-25 02:55:13,774 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5., storeName=9d5b411b5fe67e5bb007f4d36e3e11e5/A, priority=12, startTime=1682391313274; duration=0sec 2023-04-25 02:55:13,774 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 02:55:13,798 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=180 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/.tmp/B/f2fd638cc44b47939bbbc6e7dbea7fcd 2023-04-25 02:55:13,845 DEBUG [Thread-2260] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:55:13,845 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:55:13,847 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:55:13,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:55:13,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:55:13,848 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:55:13,848 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:13,848 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:55:13,848 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:55:13,848 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:13,848 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:13,849 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:13,849 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:13,849 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:13,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:13,850 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:13,850 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:55:13,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:13,850 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:13,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:55:13,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:13,850 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:55:13,851 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:55:13,851 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:55:13,851 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:55:13,851 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:13,851 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:55:13,851 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:55:13,851 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:55:13,851 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. 2023-04-25 02:55:13,851 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. started... 2023-04-25 02:55:13,851 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. as already flushing 2023-04-25 02:55:13,851 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. 2023-04-25 02:55:13,851 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:55:13,851 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:55:13,851 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:13,851 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:55:13,851 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:55:13,856 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:13,856 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:13,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:13,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:13,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:13,857 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:13,857 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:55:13,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:13,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:13,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:13,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:13,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:13,858 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:55:13,858 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@437ca62a[Count = 0] remaining members to acquire global barrier 2023-04-25 02:55:13,858 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:55:13,858 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:13,859 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:13,859 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:13,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:13,860 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:55:13,860 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:55:13,860 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:55:13,860 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:13,860 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:55:13,861 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:55:13,861 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:13,861 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:13,861 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:55:13,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:13,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:13,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:13,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:13,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:13,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:13,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:13,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:13,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:13,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:13,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:55:13,863 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:55:13,863 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:55:13,863 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:55:13,863 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:55:13,864 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:55:13,864 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:55:13,865 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:13,865 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:13,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:13,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:13,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:13,865 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:13,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:13,865 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:13,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:13,865 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:13,865 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:13,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:13,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:13,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:13,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:13,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:13,866 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:13,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:13,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:13,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:13,869 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:13,869 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:13,869 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:13,869 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:13,869 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:13,869 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:13,869 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:13,869 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:13,869 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:13,869 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:55:13,869 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:55:13,869 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:13,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:13,869 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:13,869 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:55:13,870 DEBUG [Thread-2260] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:55:13,870 DEBUG [Thread-2260] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:55:13,870 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:13,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:13,970 DEBUG [Thread-2260] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:55:13,970 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:55:13,972 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:55:13,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:55:13,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:55:13,973 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:55:13,973 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:13,973 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:55:13,973 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:55:13,974 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:13,974 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:13,975 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:13,975 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:13,975 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:13,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:13,975 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:13,975 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:55:13,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:13,976 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:13,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:55:13,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:13,976 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:55:13,976 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:55:13,976 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:55:13,976 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:55:13,976 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:13,976 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:55:13,976 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:55:13,976 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:55:13,977 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. 2023-04-25 02:55:13,977 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. started... 2023-04-25 02:55:13,977 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. as already flushing 2023-04-25 02:55:13,977 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. 2023-04-25 02:55:13,977 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:55:13,977 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:55:13,977 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:13,977 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:55:13,977 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:55:13,978 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:13,978 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:13,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:13,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:13,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:13,978 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:13,978 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:55:13,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:13,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:13,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:13,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:13,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:13,980 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:55:13,980 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:55:13,980 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1a2bceb5[Count = 0] remaining members to acquire global barrier 2023-04-25 02:55:13,980 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:13,981 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:13,981 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:13,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:13,981 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:55:13,981 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:55:13,981 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:55:13,981 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:13,981 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:55:13,983 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:13,983 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:55:13,983 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:13,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:13,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:13,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:13,983 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:55:13,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:13,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:13,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:13,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:13,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:13,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:13,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:13,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:55:13,986 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:55:13,986 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:55:13,986 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:55:13,986 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:55:13,986 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:55:13,986 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:55:13,987 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:13,987 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:13,987 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:13,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:13,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:13,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:13,987 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:13,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:13,988 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:13,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:13,988 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:13,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:13,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:13,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:13,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:13,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:13,989 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:13,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:13,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:13,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:13,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:13,991 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:13,991 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:13,991 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:13,991 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:13,991 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:13,991 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:13,991 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:13,991 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:55:13,991 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:55:13,991 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:13,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:13,991 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:55:13,991 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:13,991 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:13,991 DEBUG [Thread-2260] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:55:13,991 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:13,991 DEBUG [Thread-2260] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:55:13,992 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:13,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:14,009 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:14,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54884 deadline: 1682391374006, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:14,015 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:14,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54924 deadline: 1682391374012, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:14,092 DEBUG [Thread-2260] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:55:14,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:55:14,093 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:55:14,094 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:55:14,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:55:14,095 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:55:14,095 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:14,095 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:55:14,095 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:55:14,095 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:14,095 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:14,097 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:14,097 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:14,097 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:14,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:14,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:14,097 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:14,097 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:55:14,097 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:14,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:55:14,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:14,098 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:55:14,098 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:55:14,098 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:55:14,098 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:55:14,098 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:14,098 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:55:14,098 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:55:14,098 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:55:14,098 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. 2023-04-25 02:55:14,099 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. started... 2023-04-25 02:55:14,099 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. as already flushing 2023-04-25 02:55:14,099 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. 2023-04-25 02:55:14,099 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:55:14,099 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:55:14,099 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:14,099 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:55:14,099 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:55:14,101 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:14,101 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:14,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:14,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:14,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:14,102 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:14,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:14,102 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:55:14,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:14,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:14,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:14,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:14,103 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:55:14,103 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@48ad5785[Count = 0] remaining members to acquire global barrier 2023-04-25 02:55:14,103 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:55:14,103 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:14,104 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:14,104 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:14,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:14,104 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:55:14,104 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:55:14,104 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:14,104 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:55:14,104 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:55:14,105 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:14,106 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:55:14,106 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:14,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:14,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:14,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:14,106 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:55:14,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:14,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:14,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:14,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:14,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:14,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:14,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:14,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:55:14,108 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:55:14,108 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:55:14,108 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:55:14,108 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:55:14,108 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:55:14,108 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:55:14,112 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:14,112 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:14,112 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:14,112 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:14,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:14,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:14,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:14,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:14,113 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:14,113 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:14,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:14,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:14,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:14,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:14,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:14,114 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:14,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:14,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:14,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:14,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:14,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:14,119 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:14,119 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:14,119 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:14,119 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:14,119 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:14,119 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:55:14,119 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:55:14,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:14,119 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:14,119 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:14,119 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:14,120 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:14,120 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:14,120 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:14,119 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:55:14,120 DEBUG [Thread-2260] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:55:14,120 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:14,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:14,120 DEBUG [Thread-2260] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:55:14,185 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:14,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54940 deadline: 1682391374182, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:14,191 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:14,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54886 deadline: 1682391374189, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:14,205 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:14,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54894 deadline: 1682391374202, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:14,211 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=180 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/.tmp/C/5fc6dab5c0694ae0bea08472cd197238 2023-04-25 02:55:14,215 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/.tmp/A/a15c8b364906459f9302cbcc776860f8 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/A/a15c8b364906459f9302cbcc776860f8 2023-04-25 02:55:14,218 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/A/a15c8b364906459f9302cbcc776860f8, entries=150, sequenceid=180, filesize=25.3 K 2023-04-25 02:55:14,219 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/.tmp/B/f2fd638cc44b47939bbbc6e7dbea7fcd as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/B/f2fd638cc44b47939bbbc6e7dbea7fcd 2023-04-25 02:55:14,220 DEBUG [Thread-2260] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:55:14,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:55:14,223 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/B/f2fd638cc44b47939bbbc6e7dbea7fcd, entries=150, sequenceid=180, filesize=11.8 K 2023-04-25 02:55:14,224 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:55:14,224 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/.tmp/C/5fc6dab5c0694ae0bea08472cd197238 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/C/5fc6dab5c0694ae0bea08472cd197238 2023-04-25 02:55:14,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:55:14,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:55:14,226 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:55:14,226 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:14,226 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:55:14,226 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:55:14,231 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:14,231 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:14,233 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:14,233 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:14,233 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:14,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:14,233 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/C/5fc6dab5c0694ae0bea08472cd197238, entries=150, sequenceid=180, filesize=11.8 K 2023-04-25 02:55:14,234 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 9d5b411b5fe67e5bb007f4d36e3e11e5 in 904ms, sequenceid=180, compaction requested=false 2023-04-25 02:55:14,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:14,234 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:14,234 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:55:14,234 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 9d5b411b5fe67e5bb007f4d36e3e11e5: 2023-04-25 02:55:14,234 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:14,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:55:14,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:14,234 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:55:14,235 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:55:14,235 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:55:14,235 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:55:14,235 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:14,235 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:55:14,235 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:55:14,235 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:55:14,235 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. 2023-04-25 02:55:14,235 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. started... 2023-04-25 02:55:14,235 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 9d5b411b5fe67e5bb007f4d36e3e11e5 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-25 02:55:14,235 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9d5b411b5fe67e5bb007f4d36e3e11e5, store=A 2023-04-25 02:55:14,235 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:55:14,235 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9d5b411b5fe67e5bb007f4d36e3e11e5, store=B 2023-04-25 02:55:14,235 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:55:14,236 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9d5b411b5fe67e5bb007f4d36e3e11e5, store=C 2023-04-25 02:55:14,236 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:55:14,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(9158): Flush requested on 9d5b411b5fe67e5bb007f4d36e3e11e5 2023-04-25 02:55:14,518 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. as already flushing 2023-04-25 02:55:14,577 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:14,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54884 deadline: 1682391374572, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:14,582 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:14,582 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54924 deadline: 1682391374573, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:14,654 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304257fd2ca48b8c046c1a27937db09ade548 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304257fd2ca48b8c046c1a27937db09ade548 2023-04-25 02:55:14,659 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=195, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/.tmp/A/78b7d87538f14c738296e7cadb416c9e 2023-04-25 02:55:14,680 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:14,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54884 deadline: 1682391374678, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:14,688 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:14,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54924 deadline: 1682391374683, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:14,884 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:14,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54884 deadline: 1682391374881, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:14,894 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:14,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54924 deadline: 1682391374890, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:15,068 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=195 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/.tmp/B/66c99592f17d4d0ca68c50099d8196ad 2023-04-25 02:55:15,085 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=195 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/.tmp/C/624fed9eeb754dfcb5b9065b5d56c272 2023-04-25 02:55:15,089 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/.tmp/A/78b7d87538f14c738296e7cadb416c9e as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/A/78b7d87538f14c738296e7cadb416c9e 2023-04-25 02:55:15,092 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/A/78b7d87538f14c738296e7cadb416c9e, entries=150, sequenceid=195, filesize=25.3 K 2023-04-25 02:55:15,093 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/.tmp/B/66c99592f17d4d0ca68c50099d8196ad as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/B/66c99592f17d4d0ca68c50099d8196ad 2023-04-25 02:55:15,097 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/B/66c99592f17d4d0ca68c50099d8196ad, entries=150, sequenceid=195, filesize=11.8 K 2023-04-25 02:55:15,097 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/.tmp/C/624fed9eeb754dfcb5b9065b5d56c272 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/C/624fed9eeb754dfcb5b9065b5d56c272 2023-04-25 02:55:15,101 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/C/624fed9eeb754dfcb5b9065b5d56c272, entries=150, sequenceid=195, filesize=11.8 K 2023-04-25 02:55:15,101 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 9d5b411b5fe67e5bb007f4d36e3e11e5 in 866ms, sequenceid=195, compaction requested=true 2023-04-25 02:55:15,101 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9d5b411b5fe67e5bb007f4d36e3e11e5: 2023-04-25 02:55:15,101 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. 2023-04-25 02:55:15,102 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:55:15,102 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:55:15,102 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:15,102 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:55:15,102 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:55:15,104 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:15,104 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:15,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:15,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:15,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:15,105 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:15,105 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:55:15,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:15,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:15,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:15,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:15,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:15,107 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:55:15,108 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7bc28cd2[Count = 0] remaining members to acquire global barrier 2023-04-25 02:55:15,108 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:55:15,108 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:15,109 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:15,109 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:15,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:15,109 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:55:15,109 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:55:15,109 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:55:15,109 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:15,109 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:55:15,110 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:55:15,110 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:15,110 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:55:15,111 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:15,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:15,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:15,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:15,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:15,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:15,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:15,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:15,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:15,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:15,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:15,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:55:15,114 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:55:15,114 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:55:15,114 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:55:15,115 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:55:15,115 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:55:15,115 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:55:15,119 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:15,119 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:15,119 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:15,119 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:15,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:15,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:15,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:15,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:15,119 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:15,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:15,120 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:15,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:15,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:15,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:15,121 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:15,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:15,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:15,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:15,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:15,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:15,129 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:15,129 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:15,129 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:15,129 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:15,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:15,129 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:55:15,129 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:55:15,129 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:15,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:15,129 DEBUG [Thread-2260] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:55:15,129 DEBUG [Thread-2260] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:55:15,129 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:55:15,131 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:15,131 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:15,131 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:15,131 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:15,131 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:15,131 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:15,131 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:15,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(9158): Flush requested on 9d5b411b5fe67e5bb007f4d36e3e11e5 2023-04-25 02:55:15,191 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682391314548, max timestamp=1682391315189Num uniques 859; 2023-04-25 02:55:15,191 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682391314548, max timestamp=1682391315189Num uniques 859; 2023-04-25 02:55:15,191 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682391314548, max timestamp=1682391315189Num uniques 859; 2023-04-25 02:55:15,191 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 9d5b411b5fe67e5bb007f4d36e3e11e5 3/3 column families, dataSize=58.14 KB heapSize=164.17 KB 2023-04-25 02:55:15,191 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-25 02:55:15,192 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-25 02:55:15,191 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-25 02:55:15,192 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9d5b411b5fe67e5bb007f4d36e3e11e5, store=A 2023-04-25 02:55:15,192 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 02:55:15,192 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9d5b411b5fe67e5bb007f4d36e3e11e5, store=B 2023-04-25 02:55:15,192 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 02:55:15,192 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9d5b411b5fe67e5bb007f4d36e3e11e5, store=C 2023-04-25 02:55:15,192 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 02:55:15,207 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230425192ab37a9d824ced91e05413c8a1fef6 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425192ab37a9d824ced91e05413c8a1fef6 2023-04-25 02:55:15,229 DEBUG [Thread-2260] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:55:15,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:55:15,231 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:55:15,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:55:15,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:55:15,233 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:55:15,233 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:15,233 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:55:15,233 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:55:15,233 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:15,233 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:15,235 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:15,235 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:15,235 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:15,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:15,235 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:15,235 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:55:15,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:15,236 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:15,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:55:15,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:15,236 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:55:15,236 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:55:15,236 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:55:15,236 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:55:15,236 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:15,236 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:55:15,236 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:55:15,237 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:55:15,237 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. 2023-04-25 02:55:15,237 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. started... 2023-04-25 02:55:15,237 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. as already flushing 2023-04-25 02:55:15,237 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. 2023-04-25 02:55:15,237 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:55:15,237 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:55:15,237 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:15,237 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:55:15,237 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:55:15,238 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:15,238 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:15,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:15,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:15,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:15,238 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:15,238 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:55:15,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:15,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:15,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:15,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:15,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:15,241 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:55:15,241 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5d292876[Count = 0] remaining members to acquire global barrier 2023-04-25 02:55:15,241 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:55:15,241 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:15,242 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:15,243 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:15,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:15,243 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:55:15,243 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:15,243 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:55:15,243 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:55:15,243 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:55:15,245 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:15,245 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:55:15,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:15,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:15,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:15,245 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:15,245 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:55:15,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:15,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:15,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:15,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:15,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:15,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:15,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:15,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:55:15,248 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:55:15,248 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:55:15,248 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:55:15,248 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:55:15,248 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:55:15,248 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:55:15,250 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:15,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:15,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:15,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:15,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:15,250 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:15,251 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:15,251 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:15,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:15,251 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:15,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:15,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:15,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:15,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:15,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:15,252 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:15,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:15,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:15,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:15,253 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:15,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:15,255 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:15,255 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:15,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:15,256 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:15,256 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:15,256 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:55:15,256 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:55:15,256 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:15,256 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:15,257 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:15,257 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:15,257 DEBUG [Thread-2260] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:55:15,257 DEBUG [Thread-2260] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:55:15,257 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:15,257 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:15,257 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:15,257 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:15,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:15,258 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682391315239, max timestamp=1682391315256Num uniques 859; 2023-04-25 02:55:15,258 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-25 02:55:15,258 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682391315239, max timestamp=1682391315256Num uniques 859; 2023-04-25 02:55:15,258 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-25 02:55:15,259 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682391315239, max timestamp=1682391315256Num uniques 859; 2023-04-25 02:55:15,259 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-25 02:55:15,337 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1682391315308, max timestamp=1682391315334Num uniques 1068; 2023-04-25 02:55:15,337 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1682391315308, max timestamp=1682391315334Num uniques 1068; 2023-04-25 02:55:15,337 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1682391315308, max timestamp=1682391315334Num uniques 1068; 2023-04-25 02:55:15,337 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=7688, suffix heap size=139040,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=168 2023-04-25 02:55:15,337 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=7688, suffix heap size=139040,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=168 2023-04-25 02:55:15,337 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=7688, suffix heap size=139040,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=168 2023-04-25 02:55:15,357 DEBUG [Thread-2260] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:55:15,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:55:15,359 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:55:15,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:55:15,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:55:15,360 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:55:15,360 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:15,360 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:55:15,360 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:55:15,361 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:15,361 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:15,362 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:15,362 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:15,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:15,362 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:15,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:15,362 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:15,362 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:55:15,363 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:15,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:55:15,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:15,363 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:55:15,363 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:55:15,363 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:55:15,363 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:55:15,363 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:15,363 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:55:15,364 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:55:15,364 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:55:15,364 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. 2023-04-25 02:55:15,364 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. started... 2023-04-25 02:55:15,364 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. as already flushing 2023-04-25 02:55:15,364 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. 2023-04-25 02:55:15,364 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:55:15,364 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:55:15,364 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:15,364 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:55:15,364 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:55:15,366 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:15,366 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:15,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:15,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:15,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:15,366 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:15,366 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:55:15,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:15,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:15,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:15,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:15,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:15,368 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:55:15,368 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@19b31b00[Count = 0] remaining members to acquire global barrier 2023-04-25 02:55:15,368 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:55:15,368 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:15,369 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:15,369 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:15,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:15,369 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:55:15,369 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:55:15,369 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:55:15,369 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:15,369 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:55:15,370 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:15,370 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:55:15,370 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:15,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:15,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:15,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:15,370 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:55:15,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:15,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:15,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:15,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:15,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:15,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:15,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:15,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:55:15,373 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:55:15,373 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:55:15,373 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:55:15,373 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:55:15,373 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:55:15,373 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:55:15,374 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:15,374 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:15,375 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:15,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:15,375 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:15,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:15,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:15,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:15,375 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:15,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:15,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:15,375 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:15,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:15,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:15,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:15,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:15,376 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:15,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:15,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:15,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:15,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:15,379 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:15,380 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:15,380 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:15,380 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:15,380 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:15,380 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:15,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:15,380 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:15,380 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:15,380 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:55:15,380 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:55:15,380 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:15,380 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:15,380 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:15,380 DEBUG [Thread-2260] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:55:15,380 DEBUG [Thread-2260] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:55:15,380 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:15,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:15,411 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1682391315379, max timestamp=1682391315409Num uniques 1027; 2023-04-25 02:55:15,412 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=10392, suffix heap size=134776,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=227 2023-04-25 02:55:15,412 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1682391315379, max timestamp=1682391315409Num uniques 1027; 2023-04-25 02:55:15,412 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=10392, suffix heap size=134776,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=227 2023-04-25 02:55:15,412 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1682391315379, max timestamp=1682391315409Num uniques 1027; 2023-04-25 02:55:15,412 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=10392, suffix heap size=134776,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=227 2023-04-25 02:55:15,480 DEBUG [Thread-2260] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:55:15,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:55:15,483 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1682391315466, max timestamp=1682391315481Num uniques 1086; 2023-04-25 02:55:15,484 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=10806, suffix heap size=140912,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=236 2023-04-25 02:55:15,484 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1682391315466, max timestamp=1682391315481Num uniques 1086; 2023-04-25 02:55:15,484 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1682391315466, max timestamp=1682391315481Num uniques 1086; 2023-04-25 02:55:15,484 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=10806, suffix heap size=140912,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=236 2023-04-25 02:55:15,484 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=10806, suffix heap size=140912,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=236 2023-04-25 02:55:15,485 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:55:15,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:55:15,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:55:15,486 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:55:15,486 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:15,486 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:55:15,486 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:55:15,486 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:15,487 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:15,488 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:15,488 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:15,488 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:15,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:15,488 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:15,488 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:55:15,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:15,488 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:15,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:55:15,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:15,489 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:55:15,489 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:55:15,489 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:55:15,489 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:55:15,489 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:15,489 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:55:15,489 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:55:15,489 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:55:15,489 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. 2023-04-25 02:55:15,489 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. started... 2023-04-25 02:55:15,489 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. as already flushing 2023-04-25 02:55:15,489 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. 2023-04-25 02:55:15,489 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:55:15,489 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:55:15,490 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:15,490 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:55:15,490 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:55:15,491 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:15,491 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:15,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:15,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:15,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:15,491 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:15,491 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:55:15,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:15,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:15,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:15,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:15,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:15,493 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:55:15,493 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3b0066cc[Count = 0] remaining members to acquire global barrier 2023-04-25 02:55:15,493 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:55:15,493 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:15,494 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:15,494 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:15,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:15,494 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:55:15,494 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:55:15,494 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:55:15,494 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:15,494 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:55:15,497 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:15,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:15,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:15,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:15,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:15,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:15,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:15,498 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:55:15,498 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:15,498 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:55:15,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:15,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:15,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:15,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:15,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:55:15,500 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:55:15,500 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:55:15,500 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:55:15,500 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:55:15,500 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:55:15,500 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:55:15,501 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:15,501 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:15,501 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:15,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:15,501 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:15,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:15,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:15,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:15,502 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:15,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:15,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:15,502 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:15,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:15,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:15,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:15,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:15,503 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:15,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:15,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:15,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:15,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:15,505 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:15,505 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:15,505 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:15,505 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:55:15,505 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:55:15,505 DEBUG [Thread-2260] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:55:15,505 DEBUG [Thread-2260] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:55:15,505 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:15,505 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:15,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:15,506 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:15,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:15,505 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:15,507 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:15,507 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:15,507 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:15,507 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:15,507 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:15,543 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1682391315522, max timestamp=1682391315540Num uniques 1095; 2023-04-25 02:55:15,543 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=8930, suffix heap size=141848,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=195 2023-04-25 02:55:15,543 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1682391315522, max timestamp=1682391315540Num uniques 1095; 2023-04-25 02:55:15,543 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=8930, suffix heap size=141848,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=195 2023-04-25 02:55:15,543 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1682391315522, max timestamp=1682391315540Num uniques 1095; 2023-04-25 02:55:15,543 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=8930, suffix heap size=141848,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=195 2023-04-25 02:55:15,603 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=254, cellSize=11630, totalHeapSize=26656, min timestamp=1682391315560, max timestamp=1682391315601Num uniques 1054; 2023-04-25 02:55:15,603 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=11630, suffix heap size=137584,new segment heap size=26656 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=254 2023-04-25 02:55:15,604 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=254, cellSize=11630, totalHeapSize=26656, min timestamp=1682391315560, max timestamp=1682391315601Num uniques 1054; 2023-04-25 02:55:15,604 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=11630, suffix heap size=137584,new segment heap size=26656 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=254 2023-04-25 02:55:15,604 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=254, cellSize=11630, totalHeapSize=26656, min timestamp=1682391315560, max timestamp=1682391315601Num uniques 1054; 2023-04-25 02:55:15,604 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=11630, suffix heap size=137584,new segment heap size=26656 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=254 2023-04-25 02:55:15,605 DEBUG [Thread-2260] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:55:15,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:55:15,607 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:55:15,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:55:15,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:55:15,609 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:55:15,609 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:15,609 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:55:15,609 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:55:15,609 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:15,609 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:15,611 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:15,611 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:15,611 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:15,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:15,611 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:15,611 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:55:15,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:15,611 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:15,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:55:15,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:15,612 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:55:15,612 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:55:15,612 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:55:15,612 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:55:15,612 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:15,612 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:55:15,612 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:55:15,612 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:55:15,612 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. 2023-04-25 02:55:15,613 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. started... 2023-04-25 02:55:15,613 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=216, memsize=8.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/.tmp/A/8cc484f08b9d4fbb92fa75c9c9de144c 2023-04-25 02:55:15,613 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. as already flushing 2023-04-25 02:55:15,613 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. 2023-04-25 02:55:15,613 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:55:15,613 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:55:15,613 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:15,613 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:55:15,613 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:55:15,615 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:15,615 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:15,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:15,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:15,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:15,616 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:15,616 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:55:15,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:15,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:15,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:15,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:15,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:15,617 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:55:15,617 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@544bd962[Count = 0] remaining members to acquire global barrier 2023-04-25 02:55:15,617 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:55:15,617 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:15,619 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:15,619 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:55:15,619 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:15,619 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:15,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:15,619 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:55:15,619 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:55:15,619 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:55:15,620 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:55:15,620 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:15,621 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:55:15,621 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:15,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:15,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:15,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:15,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:15,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:15,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:15,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:15,623 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=216 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/.tmp/B/c51b3153725541ca879c6a927da493eb 2023-04-25 02:55:15,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:15,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:15,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:15,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:55:15,625 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:55:15,625 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:55:15,625 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:55:15,625 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:55:15,625 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:55:15,625 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:55:15,627 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:15,627 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:15,627 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:15,627 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:15,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:15,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:15,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:15,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:15,628 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:15,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:15,628 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:15,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:15,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:15,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:15,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:15,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:15,629 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:15,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:15,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:15,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:15,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:15,632 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:15,632 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:15,632 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:15,632 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:15,632 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:15,632 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:55:15,632 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:15,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:15,632 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:15,632 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:15,633 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:15,633 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:15,632 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:55:15,632 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:15,633 DEBUG [Thread-2260] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:55:15,633 DEBUG [Thread-2260] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:55:15,633 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:15,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:15,634 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:55:15,683 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1682391315663, max timestamp=1682391315681Num uniques 1113; 2023-04-25 02:55:15,683 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=50968, new segment data size=7458, suffix heap size=143720,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1113, new segment cells count=163 2023-04-25 02:55:15,684 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1682391315663, max timestamp=1682391315681Num uniques 1113; 2023-04-25 02:55:15,684 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=50968, new segment data size=7458, suffix heap size=143720,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1113, new segment cells count=163 2023-04-25 02:55:15,684 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1682391315663, max timestamp=1682391315681Num uniques 1113; 2023-04-25 02:55:15,684 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=50968, new segment data size=7458, suffix heap size=143720,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1113, new segment cells count=163 2023-04-25 02:55:15,733 DEBUG [Thread-2260] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:55:15,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:55:15,736 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:55:15,737 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10162, totalHeapSize=23328, min timestamp=1682391315713, max timestamp=1682391315735Num uniques 1022; 2023-04-25 02:55:15,737 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=10162, suffix heap size=134256,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=222 2023-04-25 02:55:15,737 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10162, totalHeapSize=23328, min timestamp=1682391315713, max timestamp=1682391315735Num uniques 1022; 2023-04-25 02:55:15,737 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10162, totalHeapSize=23328, min timestamp=1682391315713, max timestamp=1682391315735Num uniques 1022; 2023-04-25 02:55:15,737 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=10162, suffix heap size=134256,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=222 2023-04-25 02:55:15,737 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=10162, suffix heap size=134256,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=222 2023-04-25 02:55:15,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:55:15,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:55:15,738 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:55:15,738 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:15,738 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:55:15,738 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:55:15,738 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:15,738 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:15,739 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:15,739 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:15,739 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:15,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:15,740 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:15,740 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:55:15,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:15,740 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:15,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:55:15,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:15,740 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:55:15,741 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:55:15,741 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:55:15,741 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:55:15,741 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:15,741 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:55:15,741 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:55:15,741 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:55:15,741 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. 2023-04-25 02:55:15,741 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. started... 2023-04-25 02:55:15,741 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. as already flushing 2023-04-25 02:55:15,741 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. 2023-04-25 02:55:15,741 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:55:15,741 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:55:15,741 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:15,742 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:55:15,742 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:55:15,743 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:15,743 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:15,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:15,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:15,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:15,743 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:15,743 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:55:15,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:15,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:15,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:15,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:15,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:15,745 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:55:15,745 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@40353105[Count = 0] remaining members to acquire global barrier 2023-04-25 02:55:15,745 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:55:15,745 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:15,746 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:15,746 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:15,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:15,746 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:55:15,746 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:55:15,747 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:55:15,746 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:15,747 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:55:15,748 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:55:15,749 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:15,748 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:15,749 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:55:15,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:15,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:15,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:15,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:15,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:15,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:15,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:15,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:15,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:15,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:15,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:55:15,751 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:55:15,751 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:55:15,751 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:55:15,751 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:55:15,751 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:55:15,751 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:55:15,752 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:15,752 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:15,753 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:15,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:15,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:15,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:15,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:15,753 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:15,753 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:15,753 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:15,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:15,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:15,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:15,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:15,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:15,754 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:15,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:15,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:15,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:15,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:15,756 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:15,756 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:15,756 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:15,756 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:55:15,756 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:55:15,756 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:15,757 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:15,757 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:15,757 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:15,757 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:15,757 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:15,756 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:15,757 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:15,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:15,757 DEBUG [Thread-2260] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:55:15,757 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:55:15,757 DEBUG [Thread-2260] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:55:15,757 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:15,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:15,794 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=181, cellSize=8286, totalHeapSize=19064, min timestamp=1682391315775, max timestamp=1682391315793Num uniques 1081; 2023-04-25 02:55:15,794 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=49506, new segment data size=8286, suffix heap size=140392,new segment heap size=19064 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=181 2023-04-25 02:55:15,795 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=181, cellSize=8286, totalHeapSize=19064, min timestamp=1682391315775, max timestamp=1682391315793Num uniques 1081; 2023-04-25 02:55:15,795 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=49506, new segment data size=8286, suffix heap size=140392,new segment heap size=19064 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=181 2023-04-25 02:55:15,795 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=181, cellSize=8286, totalHeapSize=19064, min timestamp=1682391315775, max timestamp=1682391315793Num uniques 1081; 2023-04-25 02:55:15,795 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=49506, new segment data size=8286, suffix heap size=140392,new segment heap size=19064 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=181 2023-04-25 02:55:15,853 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=290, cellSize=13280, totalHeapSize=30400, min timestamp=1682391315815, max timestamp=1682391315851Num uniques 1040; 2023-04-25 02:55:15,853 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=47630, new segment data size=13280, suffix heap size=136128,new segment heap size=30400 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1040, new segment cells count=290 2023-04-25 02:55:15,854 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=290, cellSize=13280, totalHeapSize=30400, min timestamp=1682391315815, max timestamp=1682391315851Num uniques 1040; 2023-04-25 02:55:15,854 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=47630, new segment data size=13280, suffix heap size=136128,new segment heap size=30400 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1040, new segment cells count=290 2023-04-25 02:55:15,854 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=290, cellSize=13280, totalHeapSize=30400, min timestamp=1682391315815, max timestamp=1682391315851Num uniques 1040; 2023-04-25 02:55:15,854 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=47630, new segment data size=13280, suffix heap size=136128,new segment heap size=30400 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1040, new segment cells count=290 2023-04-25 02:55:15,857 DEBUG [Thread-2260] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:55:15,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:55:15,859 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:55:15,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:55:15,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:55:15,860 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:55:15,860 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:15,860 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:55:15,860 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:55:15,861 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:15,861 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:15,865 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:15,865 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:15,865 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:15,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:15,865 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:15,865 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:55:15,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:15,866 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:15,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:55:15,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:15,866 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:55:15,866 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:55:15,866 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:55:15,866 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:55:15,866 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:15,866 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:55:15,867 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:55:15,867 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:55:15,867 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. 2023-04-25 02:55:15,867 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. started... 2023-04-25 02:55:15,867 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. as already flushing 2023-04-25 02:55:15,867 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. 2023-04-25 02:55:15,867 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:55:15,867 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:55:15,867 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:15,867 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:55:15,867 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:55:15,868 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:15,868 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:15,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:15,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:15,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:15,869 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:15,869 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:55:15,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:15,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:15,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:15,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:15,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:15,870 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:55:15,870 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@415f5918[Count = 0] remaining members to acquire global barrier 2023-04-25 02:55:15,870 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:55:15,870 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:15,871 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:15,871 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:15,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:15,871 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:55:15,871 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:55:15,871 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:15,871 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:55:15,871 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:55:15,873 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:55:15,873 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:15,873 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:15,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:15,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:15,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:15,873 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:55:15,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:15,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:15,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:15,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:15,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:15,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:15,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:15,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:55:15,875 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:55:15,875 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:55:15,875 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:55:15,875 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:55:15,875 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:55:15,875 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:55:15,878 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:15,878 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:15,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:15,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:15,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:15,878 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:15,878 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:15,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:15,878 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:15,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:15,878 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:15,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:15,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:15,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:15,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:15,879 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:15,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:15,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:15,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:15,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:15,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:15,881 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:15,881 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:15,881 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:15,881 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:15,881 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:15,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:15,881 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:15,881 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:15,881 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:15,881 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:15,881 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:15,881 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:15,881 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:55:15,881 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:55:15,882 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:55:15,882 DEBUG [Thread-2260] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:55:15,882 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:15,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:15,882 DEBUG [Thread-2260] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:55:15,939 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=249, cellSize=11404, totalHeapSize=26136, min timestamp=1682391315913, max timestamp=1682391315936Num uniques 1149; 2023-04-25 02:55:15,939 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=52624, new segment data size=11404, suffix heap size=147464,new segment heap size=26136 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1149, new segment cells count=249 2023-04-25 02:55:15,939 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=249, cellSize=11404, totalHeapSize=26136, min timestamp=1682391315913, max timestamp=1682391315936Num uniques 1149; 2023-04-25 02:55:15,939 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=249, cellSize=11404, totalHeapSize=26136, min timestamp=1682391315913, max timestamp=1682391315936Num uniques 1149; 2023-04-25 02:55:15,939 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=52624, new segment data size=11404, suffix heap size=147464,new segment heap size=26136 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1149, new segment cells count=249 2023-04-25 02:55:15,939 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=52624, new segment data size=11404, suffix heap size=147464,new segment heap size=26136 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1149, new segment cells count=249 2023-04-25 02:55:15,982 DEBUG [Thread-2260] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:55:15,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:55:15,984 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:55:15,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:55:15,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:55:15,985 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:55:15,985 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:15,985 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:55:15,985 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:55:15,985 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:15,985 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:15,986 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:15,986 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:15,986 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:15,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:15,987 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:15,987 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:55:15,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:15,987 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:15,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:55:15,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:15,987 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:55:15,993 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:55:15,993 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:55:15,994 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:55:15,994 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:15,994 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:55:15,994 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:55:15,994 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:55:15,994 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. 2023-04-25 02:55:15,994 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. started... 2023-04-25 02:55:15,994 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. as already flushing 2023-04-25 02:55:15,994 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. 2023-04-25 02:55:15,994 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:55:15,994 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:55:15,994 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:15,994 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:55:15,994 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:55:15,995 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:15,995 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:15,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:15,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:15,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:15,996 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:15,996 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:55:15,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:15,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:15,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:15,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:15,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:15,997 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:55:15,997 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@795171f9[Count = 0] remaining members to acquire global barrier 2023-04-25 02:55:15,997 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:55:15,997 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:15,999 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=208, cellSize=9520, totalHeapSize=21872, min timestamp=1682391315979, max timestamp=1682391315997Num uniques 1108; 2023-04-25 02:55:15,999 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=50740, new segment data size=9520, suffix heap size=143200,new segment heap size=21872 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1108, new segment cells count=208 2023-04-25 02:55:15,999 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=208, cellSize=9520, totalHeapSize=21872, min timestamp=1682391315979, max timestamp=1682391315997Num uniques 1108; 2023-04-25 02:55:15,999 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=50740, new segment data size=9520, suffix heap size=143200,new segment heap size=21872 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1108, new segment cells count=208 2023-04-25 02:55:15,999 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:16,000 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:16,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:16,000 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:55:16,000 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:55:16,000 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:55:16,000 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:16,000 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:55:16,000 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=208, cellSize=9520, totalHeapSize=21872, min timestamp=1682391315979, max timestamp=1682391315997Num uniques 1108; 2023-04-25 02:55:16,000 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=50740, new segment data size=9520, suffix heap size=143200,new segment heap size=21872 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1108, new segment cells count=208 2023-04-25 02:55:16,001 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:55:16,001 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:16,001 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:16,001 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:55:16,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:16,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:16,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:16,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:16,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:16,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:16,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:16,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:16,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:16,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:16,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:55:16,003 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:55:16,003 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:55:16,003 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:55:16,003 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:55:16,003 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:55:16,003 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:55:16,005 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:16,005 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:16,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:16,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:16,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:16,005 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:16,005 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:16,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:16,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:16,005 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:16,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:16,005 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:16,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:16,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:16,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:16,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:16,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:16,008 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:16,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:16,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:16,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:16,010 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:16,010 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:16,010 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:16,010 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:55:16,010 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:16,010 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:16,010 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:16,010 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:16,010 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:16,010 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:16,010 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:55:16,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:16,010 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:16,010 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:16,011 DEBUG [Thread-2260] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:55:16,011 DEBUG [Thread-2260] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:55:16,011 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:16,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:16,035 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=216 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/.tmp/C/951d1d4e165143069ac4b7d85ae7b92d 2023-04-25 02:55:16,039 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/.tmp/A/8cc484f08b9d4fbb92fa75c9c9de144c as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/A/8cc484f08b9d4fbb92fa75c9c9de144c 2023-04-25 02:55:16,042 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/A/8cc484f08b9d4fbb92fa75c9c9de144c, entries=200, sequenceid=216, filesize=32.1 K 2023-04-25 02:55:16,043 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/.tmp/B/c51b3153725541ca879c6a927da493eb as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/B/c51b3153725541ca879c6a927da493eb 2023-04-25 02:55:16,047 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/B/c51b3153725541ca879c6a927da493eb, entries=150, sequenceid=216, filesize=11.8 K 2023-04-25 02:55:16,048 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/.tmp/C/951d1d4e165143069ac4b7d85ae7b92d as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/C/951d1d4e165143069ac4b7d85ae7b92d 2023-04-25 02:55:16,054 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/C/951d1d4e165143069ac4b7d85ae7b92d, entries=150, sequenceid=216, filesize=11.8 K 2023-04-25 02:55:16,055 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~66.19 KB/67776, currentSize=107.34 KB/109920 for 9d5b411b5fe67e5bb007f4d36e3e11e5 in 864ms, sequenceid=216, compaction requested=true 2023-04-25 02:55:16,055 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 9d5b411b5fe67e5bb007f4d36e3e11e5: 2023-04-25 02:55:16,055 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 02:55:16,055 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-25 02:55:16,055 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-25 02:55:16,055 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 02:55:16,056 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 02:55:16,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(9158): Flush requested on 9d5b411b5fe67e5bb007f4d36e3e11e5 2023-04-25 02:55:16,057 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 9d5b411b5fe67e5bb007f4d36e3e11e5 3/3 column families, dataSize=114.05 KB heapSize=320.63 KB 2023-04-25 02:55:16,058 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 110848 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-25 02:55:16,058 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48418 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-25 02:55:16,058 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HStore(1912): 9d5b411b5fe67e5bb007f4d36e3e11e5/A is initiating minor compaction (all files) 2023-04-25 02:55:16,058 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9d5b411b5fe67e5bb007f4d36e3e11e5, store=A 2023-04-25 02:55:16,058 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HStore(1912): 9d5b411b5fe67e5bb007f4d36e3e11e5/B is initiating minor compaction (all files) 2023-04-25 02:55:16,058 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 02:55:16,058 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 9d5b411b5fe67e5bb007f4d36e3e11e5/A in TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. 2023-04-25 02:55:16,058 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9d5b411b5fe67e5bb007f4d36e3e11e5, store=B 2023-04-25 02:55:16,058 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/A/82ca2bf000e646d9b901407e5783065f, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/A/a15c8b364906459f9302cbcc776860f8, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/A/78b7d87538f14c738296e7cadb416c9e, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/A/8cc484f08b9d4fbb92fa75c9c9de144c] into tmpdir=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/.tmp, totalSize=108.3 K 2023-04-25 02:55:16,058 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 9d5b411b5fe67e5bb007f4d36e3e11e5/B in TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. 2023-04-25 02:55:16,058 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 02:55:16,058 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/B/3703b5da4c594ed68d58845cfd8cb0ad, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/B/f2fd638cc44b47939bbbc6e7dbea7fcd, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/B/66c99592f17d4d0ca68c50099d8196ad, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/B/c51b3153725541ca879c6a927da493eb] into tmpdir=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/.tmp, totalSize=47.3 K 2023-04-25 02:55:16,059 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting 82ca2bf000e646d9b901407e5783065f, keycount=150, bloomtype=ROW, size=25.6 K, encoding=NONE, compression=NONE, seqNum=160, earliestPutTs=1682391311699 2023-04-25 02:55:16,059 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9d5b411b5fe67e5bb007f4d36e3e11e5, store=C 2023-04-25 02:55:16,059 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 02:55:16,059 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting 3703b5da4c594ed68d58845cfd8cb0ad, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=160, earliestPutTs=1682391311699 2023-04-25 02:55:16,059 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting a15c8b364906459f9302cbcc776860f8, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=180, earliestPutTs=1682391312041 2023-04-25 02:55:16,059 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting f2fd638cc44b47939bbbc6e7dbea7fcd, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=180, earliestPutTs=1682391312041 2023-04-25 02:55:16,060 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting 78b7d87538f14c738296e7cadb416c9e, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=195, earliestPutTs=1682391313345 2023-04-25 02:55:16,060 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting 66c99592f17d4d0ca68c50099d8196ad, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=195, earliestPutTs=1682391313345 2023-04-25 02:55:16,060 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting 8cc484f08b9d4fbb92fa75c9c9de144c, keycount=200, bloomtype=ROW, size=32.1 K, encoding=NONE, compression=NONE, seqNum=216, earliestPutTs=1682391314548 2023-04-25 02:55:16,061 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting c51b3153725541ca879c6a927da493eb, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=216, earliestPutTs=1682391314548 2023-04-25 02:55:16,077 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] throttle.PressureAwareThroughputController(145): 9d5b411b5fe67e5bb007f4d36e3e11e5#B#compaction#332 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 53.85 MB/second 2023-04-25 02:55:16,085 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] throttle.PressureAwareThroughputController(145): 9d5b411b5fe67e5bb007f4d36e3e11e5#A#compaction#331 average throughput is 1.10 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-25 02:55:16,108 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:16,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54884 deadline: 1682391376098, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:16,111 DEBUG [Thread-2260] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:55:16,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:55:16,113 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:55:16,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:55:16,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:55:16,115 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:55:16,115 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:16,115 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:55:16,115 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:55:16,115 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:16,115 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:16,118 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:16,118 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:16,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:16,118 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/.tmp/A/98e9d45bf15a4411ba742e90717013e8 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/A/98e9d45bf15a4411ba742e90717013e8 2023-04-25 02:55:16,119 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:16,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:16,119 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:16,119 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:55:16,119 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:16,119 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:16,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54924 deadline: 1682391376107, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:16,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:55:16,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:16,120 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:55:16,120 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:55:16,120 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:55:16,120 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:55:16,120 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:16,121 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:55:16,121 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:55:16,121 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:55:16,121 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. 2023-04-25 02:55:16,121 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. started... 2023-04-25 02:55:16,121 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. as already flushing 2023-04-25 02:55:16,121 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. 2023-04-25 02:55:16,121 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:55:16,121 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:55:16,121 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:16,121 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:55:16,121 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:55:16,123 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:16,123 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 9d5b411b5fe67e5bb007f4d36e3e11e5/A of 9d5b411b5fe67e5bb007f4d36e3e11e5 into 98e9d45bf15a4411ba742e90717013e8(size=25.7 K), total size for store is 25.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 02:55:16,123 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9d5b411b5fe67e5bb007f4d36e3e11e5: 2023-04-25 02:55:16,123 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5., storeName=9d5b411b5fe67e5bb007f4d36e3e11e5/A, priority=12, startTime=1682391316055; duration=0sec 2023-04-25 02:55:16,123 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 02:55:16,123 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-25 02:55:16,123 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:16,123 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:55:16,124 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:16,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:16,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:16,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:16,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:16,124 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48418 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-25 02:55:16,124 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HStore(1912): 9d5b411b5fe67e5bb007f4d36e3e11e5/C is initiating minor compaction (all files) 2023-04-25 02:55:16,124 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 9d5b411b5fe67e5bb007f4d36e3e11e5/C in TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. 2023-04-25 02:55:16,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:16,125 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/C/cc1f84c3c938484f9a6bede5da083ae5, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/C/5fc6dab5c0694ae0bea08472cd197238, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/C/624fed9eeb754dfcb5b9065b5d56c272, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/C/951d1d4e165143069ac4b7d85ae7b92d] into tmpdir=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/.tmp, totalSize=47.3 K 2023-04-25 02:55:16,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:16,125 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting cc1f84c3c938484f9a6bede5da083ae5, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=160, earliestPutTs=1682391311699 2023-04-25 02:55:16,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:16,125 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting 5fc6dab5c0694ae0bea08472cd197238, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=180, earliestPutTs=1682391312041 2023-04-25 02:55:16,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:16,126 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting 624fed9eeb754dfcb5b9065b5d56c272, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=195, earliestPutTs=1682391313345 2023-04-25 02:55:16,126 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:55:16,126 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@735b2778[Count = 0] remaining members to acquire global barrier 2023-04-25 02:55:16,126 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:55:16,126 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:16,126 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting 951d1d4e165143069ac4b7d85ae7b92d, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=216, earliestPutTs=1682391314548 2023-04-25 02:55:16,127 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:16,127 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:16,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:16,127 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:55:16,127 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:55:16,127 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:55:16,127 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:16,127 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:55:16,129 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:55:16,129 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:16,129 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:16,129 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:55:16,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:16,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:16,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:16,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:16,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:16,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:16,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:16,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:16,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:16,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:16,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:55:16,131 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:55:16,131 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:55:16,131 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:55:16,131 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:55:16,132 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:55:16,132 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:55:16,133 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] throttle.PressureAwareThroughputController(145): 9d5b411b5fe67e5bb007f4d36e3e11e5#C#compaction#333 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-25 02:55:16,134 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:16,134 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:16,134 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:16,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:16,135 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:16,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:16,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:16,136 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:16,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:16,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:16,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:16,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:16,137 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:16,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:16,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:16,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:16,138 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:16,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:16,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:16,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:16,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:16,140 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:16,140 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:16,140 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:16,141 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:16,141 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:16,141 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:16,141 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:16,141 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:16,141 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:55:16,141 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:55:16,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:16,141 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:16,141 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:16,141 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:16,141 DEBUG [Thread-2260] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:55:16,141 DEBUG [Thread-2260] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:55:16,141 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:16,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:16,195 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:16,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54940 deadline: 1682391376192, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:16,197 DEBUG [Thread-2254] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4139 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5., hostname=jenkins-hbase4.apache.org,46697,1682391220023, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor28.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-25 02:55:16,200 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:16,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54886 deadline: 1682391376194, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:16,202 DEBUG [Thread-2258] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4148 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5., hostname=jenkins-hbase4.apache.org,46697,1682391220023, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor28.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-25 02:55:16,212 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:16,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54884 deadline: 1682391376209, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:16,219 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:16,219 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54894 deadline: 1682391376215, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:16,221 DEBUG [Thread-2250] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4163 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5., hostname=jenkins-hbase4.apache.org,46697,1682391220023, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor28.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-25 02:55:16,230 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:16,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54924 deadline: 1682391376221, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:16,241 DEBUG [Thread-2260] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:55:16,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:55:16,243 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:55:16,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:55:16,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:55:16,244 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:55:16,244 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:16,244 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:55:16,244 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:55:16,245 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:16,245 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:16,246 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:16,246 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:16,246 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:16,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:16,246 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:16,246 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:55:16,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:16,247 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:16,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:55:16,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:16,247 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:55:16,247 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:55:16,247 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:55:16,247 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:55:16,247 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:16,248 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:55:16,248 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:55:16,248 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:55:16,248 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. 2023-04-25 02:55:16,248 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. started... 2023-04-25 02:55:16,248 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. as already flushing 2023-04-25 02:55:16,248 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. 2023-04-25 02:55:16,248 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:55:16,248 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:55:16,248 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:16,248 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:55:16,248 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:55:16,252 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:16,252 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:16,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:16,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:16,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:16,253 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:16,253 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:55:16,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:16,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:16,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:16,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:16,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:16,254 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:55:16,254 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1a8f8d0f[Count = 0] remaining members to acquire global barrier 2023-04-25 02:55:16,254 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:55:16,254 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:16,255 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:16,255 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:16,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:16,255 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:55:16,255 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:55:16,255 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:55:16,255 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:16,255 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:55:16,256 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:16,256 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:55:16,256 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:16,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:16,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:16,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:16,256 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:55:16,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:16,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:16,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:16,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:16,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:16,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:16,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:16,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:55:16,258 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:55:16,258 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:55:16,258 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:55:16,258 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:55:16,258 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:55:16,259 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:55:16,260 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:16,260 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:16,260 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:16,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:16,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:16,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:16,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:16,260 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:16,260 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:16,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:16,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:16,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:16,261 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:16,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:16,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:16,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:16,262 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:16,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:16,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:16,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:16,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:16,267 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:16,267 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:16,267 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:16,267 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:16,268 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:55:16,268 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:16,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:16,267 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:16,268 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:55:16,267 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:16,268 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:16,268 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:16,268 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:16,268 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:16,268 DEBUG [Thread-2260] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:55:16,268 DEBUG [Thread-2260] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:55:16,268 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:16,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:16,368 DEBUG [Thread-2260] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:55:16,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:55:16,370 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:55:16,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:55:16,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:55:16,372 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:55:16,372 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:16,372 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:55:16,372 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:55:16,375 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:16,375 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:16,376 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:16,377 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:16,377 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:16,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:16,377 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:16,377 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:55:16,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:16,377 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:16,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:55:16,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:16,378 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:55:16,378 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:55:16,378 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:55:16,378 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:55:16,378 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:16,378 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:55:16,378 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:55:16,378 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. 2023-04-25 02:55:16,378 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. started... 2023-04-25 02:55:16,378 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:55:16,378 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. as already flushing 2023-04-25 02:55:16,378 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. 2023-04-25 02:55:16,378 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:55:16,378 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:55:16,378 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:16,378 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:55:16,378 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:55:16,380 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:16,380 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:16,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:16,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:16,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:16,380 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:16,380 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:55:16,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:16,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:16,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:16,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:16,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:16,381 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:55:16,382 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@212380df[Count = 0] remaining members to acquire global barrier 2023-04-25 02:55:16,382 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:55:16,382 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:16,383 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:16,383 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:16,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:16,383 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:55:16,383 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:16,383 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:55:16,383 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:55:16,383 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:55:16,385 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:55:16,385 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:16,385 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:16,385 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:55:16,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:16,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:16,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:16,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:16,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:16,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:16,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:16,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:16,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:16,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:16,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:55:16,387 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:55:16,387 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:55:16,387 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:55:16,388 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:55:16,388 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:55:16,388 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:55:16,389 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:16,389 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:16,389 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:16,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:16,389 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:16,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:16,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:16,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:16,390 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:16,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:16,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:16,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:16,390 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:16,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:16,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:16,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:16,391 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:16,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:16,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:16,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:16,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:16,393 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:16,393 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:16,393 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:16,393 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:16,393 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:16,393 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:16,393 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:16,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:16,393 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:55:16,393 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:55:16,393 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:55:16,393 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:16,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:16,393 DEBUG [Thread-2260] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:55:16,393 DEBUG [Thread-2260] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:55:16,393 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:16,394 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:16,394 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:16,394 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:16,418 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:16,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54884 deadline: 1682391376414, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:16,437 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:16,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54924 deadline: 1682391376432, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:16,474 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230425ca0eb20bef6c40e0972fe84fcdbb70c1 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425ca0eb20bef6c40e0972fe84fcdbb70c1 2023-04-25 02:55:16,479 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=439, memsize=40.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/.tmp/A/d819bdc04029412bbefde1fb19bd9c7a 2023-04-25 02:55:16,492 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=439 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/.tmp/B/2626d6f8ce5447be9decb521b7efb663 2023-04-25 02:55:16,494 DEBUG [Thread-2260] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:55:16,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:55:16,497 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:55:16,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:55:16,498 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/.tmp/B/97647b20a62d4a64aa99e34eb1fda79d as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/B/97647b20a62d4a64aa99e34eb1fda79d 2023-04-25 02:55:16,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:55:16,498 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:55:16,498 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:16,498 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:55:16,498 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:55:16,499 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:16,499 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:16,500 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:16,500 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:16,500 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:16,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:16,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:16,500 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:16,501 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:55:16,501 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:16,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:55:16,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:16,501 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:55:16,504 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:55:16,504 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:55:16,504 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:55:16,504 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:16,504 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:55:16,505 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 9d5b411b5fe67e5bb007f4d36e3e11e5/B of 9d5b411b5fe67e5bb007f4d36e3e11e5 into 97647b20a62d4a64aa99e34eb1fda79d(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 02:55:16,505 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9d5b411b5fe67e5bb007f4d36e3e11e5: 2023-04-25 02:55:16,505 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5., storeName=9d5b411b5fe67e5bb007f4d36e3e11e5/B, priority=12, startTime=1682391316055; duration=0sec 2023-04-25 02:55:16,505 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 02:55:16,506 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:55:16,507 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:55:16,507 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. 2023-04-25 02:55:16,507 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. started... 2023-04-25 02:55:16,507 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. as already flushing 2023-04-25 02:55:16,507 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. 2023-04-25 02:55:16,507 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:55:16,507 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:55:16,507 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:16,507 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:55:16,507 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:55:16,510 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:16,510 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:16,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:16,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:16,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:16,510 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:16,510 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:55:16,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:16,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:16,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:16,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:16,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:16,512 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:55:16,512 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@538fe3bb[Count = 0] remaining members to acquire global barrier 2023-04-25 02:55:16,512 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:55:16,512 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:16,513 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:16,513 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:16,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:16,513 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:55:16,513 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:55:16,513 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:55:16,513 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:16,513 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:55:16,514 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:16,514 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:55:16,514 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:16,514 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:55:16,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:16,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:16,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:16,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:16,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:16,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:16,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:16,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:16,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:16,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:16,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:55:16,518 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:55:16,518 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:55:16,518 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:55:16,518 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:55:16,518 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:55:16,518 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:55:16,520 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:16,520 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:16,520 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:16,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:16,520 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:16,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:16,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:16,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:16,520 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:16,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:16,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:16,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:16,520 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:16,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:16,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:16,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:16,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:16,523 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:16,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:16,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:16,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:16,529 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:16,529 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:16,529 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:16,529 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:16,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:16,529 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:16,529 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:16,529 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:16,529 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:55:16,529 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:55:16,529 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:16,529 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:16,530 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:16,530 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:16,530 DEBUG [Thread-2260] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:55:16,530 DEBUG [Thread-2260] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:55:16,530 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:16,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:16,550 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/.tmp/C/e9523fdf37964c0ab71c8aff66e14f9b as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/C/e9523fdf37964c0ab71c8aff66e14f9b 2023-04-25 02:55:16,554 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 9d5b411b5fe67e5bb007f4d36e3e11e5/C of 9d5b411b5fe67e5bb007f4d36e3e11e5 into e9523fdf37964c0ab71c8aff66e14f9b(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 02:55:16,554 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9d5b411b5fe67e5bb007f4d36e3e11e5: 2023-04-25 02:55:16,554 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5., storeName=9d5b411b5fe67e5bb007f4d36e3e11e5/C, priority=12, startTime=1682391316056; duration=0sec 2023-04-25 02:55:16,554 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 02:55:16,630 DEBUG [Thread-2260] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:55:16,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:55:16,632 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:55:16,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:55:16,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:55:16,633 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:55:16,633 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:16,633 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:55:16,633 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:55:16,634 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:16,634 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:16,635 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:16,635 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:16,635 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:16,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:16,635 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:16,635 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:55:16,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:16,636 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:16,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:55:16,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:16,636 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:55:16,636 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:55:16,636 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:55:16,636 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:55:16,637 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:16,637 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:55:16,637 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:55:16,637 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:55:16,637 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. 2023-04-25 02:55:16,637 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. started... 2023-04-25 02:55:16,637 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. as already flushing 2023-04-25 02:55:16,637 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. 2023-04-25 02:55:16,638 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:55:16,639 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:55:16,639 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:16,639 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:55:16,639 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:55:16,640 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:16,640 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:16,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:16,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:16,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:16,640 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:16,640 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:55:16,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:16,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:16,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:16,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:16,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:16,642 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:55:16,642 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@79456e55[Count = 0] remaining members to acquire global barrier 2023-04-25 02:55:16,642 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:55:16,642 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:16,643 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:16,643 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:16,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:16,643 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:55:16,643 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:16,643 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:55:16,643 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:55:16,643 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:55:16,648 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:55:16,648 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:16,648 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:16,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:16,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:16,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:16,648 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:55:16,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:16,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:16,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:16,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:16,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:16,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:16,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:16,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:55:16,650 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:55:16,650 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:55:16,650 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:55:16,650 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:55:16,650 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:55:16,650 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:55:16,651 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:16,651 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:16,651 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:16,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:16,652 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:16,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:16,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:16,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:16,652 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:16,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:16,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:16,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:16,652 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:16,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:16,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:16,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:16,653 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:16,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:16,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:16,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:16,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:16,655 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:16,655 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:16,655 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:16,656 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:16,656 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:55:16,656 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:55:16,655 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:16,655 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:16,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:16,656 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:16,656 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:16,656 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:16,656 DEBUG [Thread-2260] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:55:16,656 DEBUG [Thread-2260] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:55:16,656 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:16,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:16,656 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:16,656 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:16,721 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:16,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54884 deadline: 1682391376720, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:16,742 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:16,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54924 deadline: 1682391376738, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:16,756 DEBUG [Thread-2260] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:55:16,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:55:16,758 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:55:16,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:55:16,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:55:16,759 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:55:16,759 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:16,759 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:55:16,759 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:55:16,760 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:16,760 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:16,761 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:16,761 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:16,761 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:16,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:16,762 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:16,762 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:55:16,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:16,762 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:16,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:55:16,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:16,762 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:55:16,763 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:55:16,763 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:55:16,763 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:55:16,763 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:16,763 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:55:16,763 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:55:16,763 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:55:16,763 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. 2023-04-25 02:55:16,763 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. started... 2023-04-25 02:55:16,763 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. as already flushing 2023-04-25 02:55:16,763 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. 2023-04-25 02:55:16,763 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:55:16,763 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:55:16,763 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:16,763 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:55:16,763 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:55:16,766 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:16,766 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:16,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:16,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:16,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:16,766 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:16,766 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:55:16,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:16,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:16,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:16,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:16,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:16,767 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:55:16,767 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1fc46387[Count = 0] remaining members to acquire global barrier 2023-04-25 02:55:16,767 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:55:16,767 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:16,768 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:16,768 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:16,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:16,769 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:55:16,769 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:55:16,769 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:55:16,769 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:16,769 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:55:16,770 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:16,770 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:55:16,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:16,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:16,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:16,770 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:16,770 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:55:16,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:16,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:16,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:16,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:16,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:16,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:16,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:16,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:55:16,772 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:55:16,772 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:55:16,772 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:55:16,772 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:55:16,772 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:55:16,772 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:55:16,774 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:16,774 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:16,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:16,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:16,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:16,774 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:16,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:16,774 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:16,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:16,774 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:16,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:16,775 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:16,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:16,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:16,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:16,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:16,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:16,776 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:16,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:16,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:16,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:16,779 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:16,779 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:16,779 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:16,779 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:55:16,779 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:55:16,779 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:16,779 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:16,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:16,779 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:16,779 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:16,779 DEBUG [Thread-2260] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:55:16,779 DEBUG [Thread-2260] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:55:16,779 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:16,779 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:16,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:16,779 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:16,780 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:16,780 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:16,879 DEBUG [Thread-2260] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:55:16,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:55:16,882 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:55:16,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:55:16,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:55:16,883 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:55:16,883 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:16,883 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:55:16,883 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:55:16,883 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:16,883 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:16,884 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:16,884 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:16,884 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:16,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:16,885 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:16,885 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:55:16,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:16,885 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:16,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:55:16,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:16,886 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:55:16,886 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:55:16,886 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:55:16,886 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:55:16,886 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:16,886 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:55:16,886 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:55:16,886 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:55:16,886 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. 2023-04-25 02:55:16,886 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. started... 2023-04-25 02:55:16,886 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. as already flushing 2023-04-25 02:55:16,886 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. 2023-04-25 02:55:16,886 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:55:16,887 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:55:16,887 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:16,887 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:55:16,887 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:55:16,888 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:16,888 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:16,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:16,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:16,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:16,888 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:16,888 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:55:16,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:16,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:16,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:16,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:16,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:16,890 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:55:16,890 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@22dc2dd8[Count = 0] remaining members to acquire global barrier 2023-04-25 02:55:16,890 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:55:16,890 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:16,891 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:16,891 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:16,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:16,891 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:16,891 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:55:16,891 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:55:16,891 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:55:16,891 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:55:16,893 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:16,893 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:55:16,893 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:16,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:16,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:16,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:16,893 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:55:16,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:16,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:16,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:16,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:16,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:16,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:16,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:16,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:55:16,896 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:55:16,896 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:55:16,896 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:55:16,896 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:55:16,896 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:55:16,896 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:55:16,897 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:16,897 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:16,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:16,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:16,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:16,898 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:16,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:16,898 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:16,898 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:16,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:16,898 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:16,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:16,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:16,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:16,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:16,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:16,899 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:16,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:16,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:16,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:16,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:16,903 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:16,903 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:16,903 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:16,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:16,903 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:16,903 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:16,903 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:16,903 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:55:16,903 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:55:16,903 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:16,903 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:16,903 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:16,903 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:16,903 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:16,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:16,903 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:16,903 DEBUG [Thread-2260] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:55:16,903 DEBUG [Thread-2260] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:55:16,919 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=439 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/.tmp/C/0a7842e5f47a403f9545049720c83770 2023-04-25 02:55:16,923 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/.tmp/A/d819bdc04029412bbefde1fb19bd9c7a as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/A/d819bdc04029412bbefde1fb19bd9c7a 2023-04-25 02:55:16,931 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/A/d819bdc04029412bbefde1fb19bd9c7a, entries=250, sequenceid=439, filesize=39.2 K 2023-04-25 02:55:16,933 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/.tmp/B/2626d6f8ce5447be9decb521b7efb663 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/B/2626d6f8ce5447be9decb521b7efb663 2023-04-25 02:55:16,938 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/B/2626d6f8ce5447be9decb521b7efb663, entries=150, sequenceid=439, filesize=11.9 K 2023-04-25 02:55:16,939 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/.tmp/C/0a7842e5f47a403f9545049720c83770 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/C/0a7842e5f47a403f9545049720c83770 2023-04-25 02:55:16,943 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/C/0a7842e5f47a403f9545049720c83770, entries=150, sequenceid=439, filesize=11.9 K 2023-04-25 02:55:16,944 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~120.76 KB/123660, heapSize ~340.50 KB/348672, currentSize=60.38 KB/61830 for 9d5b411b5fe67e5bb007f4d36e3e11e5 in 887ms, sequenceid=439, compaction requested=false 2023-04-25 02:55:16,944 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 9d5b411b5fe67e5bb007f4d36e3e11e5: 2023-04-25 02:55:17,003 DEBUG [Thread-2260] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:55:17,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:55:17,005 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:55:17,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:55:17,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:55:17,007 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:55:17,007 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:17,007 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:55:17,007 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:55:17,007 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:17,007 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:17,008 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:17,008 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:17,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:17,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:17,008 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:17,009 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:55:17,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:17,009 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:17,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:55:17,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:17,009 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:55:17,010 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:55:17,010 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:55:17,010 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:55:17,010 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:17,010 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:55:17,010 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:55:17,010 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:55:17,010 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. 2023-04-25 02:55:17,010 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. started... 2023-04-25 02:55:17,010 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 9d5b411b5fe67e5bb007f4d36e3e11e5 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-25 02:55:17,011 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9d5b411b5fe67e5bb007f4d36e3e11e5, store=A 2023-04-25 02:55:17,011 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:55:17,011 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9d5b411b5fe67e5bb007f4d36e3e11e5, store=B 2023-04-25 02:55:17,011 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:55:17,011 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9d5b411b5fe67e5bb007f4d36e3e11e5, store=C 2023-04-25 02:55:17,011 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:55:17,030 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230425879da4c53fb34c5b97aa7bdbadeed2cb to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425879da4c53fb34c5b97aa7bdbadeed2cb 2023-04-25 02:55:17,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(9158): Flush requested on 9d5b411b5fe67e5bb007f4d36e3e11e5 2023-04-25 02:55:17,228 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. as already flushing 2023-04-25 02:55:17,284 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:17,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54924 deadline: 1682391377282, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:17,288 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:17,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 208 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54884 deadline: 1682391377283, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:17,390 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:17,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54924 deadline: 1682391377385, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:17,393 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:17,393 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54884 deadline: 1682391377389, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:17,437 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=454, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/.tmp/A/6c6b168e277349f9a7c5d63214ad32a1 2023-04-25 02:55:17,593 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:17,594 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 201 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54924 deadline: 1682391377591, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:17,599 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:17,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54884 deadline: 1682391377595, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:17,846 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=454 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/.tmp/B/d317e1976c45442496ee612abf0ac5c3 2023-04-25 02:55:17,900 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:17,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54924 deadline: 1682391377896, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:17,905 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:17,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54884 deadline: 1682391377901, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:18,257 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=454 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/.tmp/C/4ba2891cb65b44c99fd2da78a0977bf8 2023-04-25 02:55:18,261 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/.tmp/A/6c6b168e277349f9a7c5d63214ad32a1 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/A/6c6b168e277349f9a7c5d63214ad32a1 2023-04-25 02:55:18,265 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/A/6c6b168e277349f9a7c5d63214ad32a1, entries=150, sequenceid=454, filesize=25.4 K 2023-04-25 02:55:18,266 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/.tmp/B/d317e1976c45442496ee612abf0ac5c3 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/B/d317e1976c45442496ee612abf0ac5c3 2023-04-25 02:55:18,270 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/B/d317e1976c45442496ee612abf0ac5c3, entries=150, sequenceid=454, filesize=11.9 K 2023-04-25 02:55:18,271 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/.tmp/C/4ba2891cb65b44c99fd2da78a0977bf8 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/C/4ba2891cb65b44c99fd2da78a0977bf8 2023-04-25 02:55:18,274 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/C/4ba2891cb65b44c99fd2da78a0977bf8, entries=150, sequenceid=454, filesize=11.9 K 2023-04-25 02:55:18,274 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 9d5b411b5fe67e5bb007f4d36e3e11e5 in 1264ms, sequenceid=454, compaction requested=true 2023-04-25 02:55:18,274 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9d5b411b5fe67e5bb007f4d36e3e11e5: 2023-04-25 02:55:18,274 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. 2023-04-25 02:55:18,275 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:55:18,275 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:55:18,275 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:18,275 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:55:18,275 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:55:18,277 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:18,277 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:18,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:18,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:18,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:18,278 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:18,278 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:55:18,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:18,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:18,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:18,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:18,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:18,279 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:55:18,279 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@e629003[Count = 0] remaining members to acquire global barrier 2023-04-25 02:55:18,279 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:55:18,279 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:18,280 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:18,280 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:18,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:18,280 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:55:18,280 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:55:18,280 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:55:18,280 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:18,280 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:55:18,281 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:18,281 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:55:18,281 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:18,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:18,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:18,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:18,281 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:55:18,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:18,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:18,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:18,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:18,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:18,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:18,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:18,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:55:18,283 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:55:18,283 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:55:18,283 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:55:18,283 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:55:18,284 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:55:18,284 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:55:18,289 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:18,289 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:18,289 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:18,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:18,289 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:18,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:18,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:18,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:18,289 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:18,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:18,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:18,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:18,290 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:18,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:18,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:18,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:18,291 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:18,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:18,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:18,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:18,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:18,292 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:18,292 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:18,292 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:18,293 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:18,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:18,293 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:55:18,293 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:18,293 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:55:18,293 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:55:18,292 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:18,293 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:18,293 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:18,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:18,293 DEBUG [Thread-2260] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:55:18,293 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:18,293 DEBUG [Thread-2260] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:55:18,293 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:18,293 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:18,293 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:18,393 DEBUG [Thread-2260] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:55:18,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:55:18,395 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:55:18,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:55:18,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:55:18,396 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:55:18,396 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:18,396 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:55:18,396 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:55:18,397 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:18,397 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:18,398 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:18,398 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:18,398 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:18,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:18,398 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:18,398 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:55:18,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:18,399 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:18,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:55:18,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:18,399 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:55:18,399 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:55:18,399 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:55:18,399 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:55:18,399 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:18,399 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:55:18,399 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:55:18,400 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. 2023-04-25 02:55:18,400 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. started... 2023-04-25 02:55:18,400 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:55:18,400 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 9d5b411b5fe67e5bb007f4d36e3e11e5 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-25 02:55:18,400 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9d5b411b5fe67e5bb007f4d36e3e11e5, store=A 2023-04-25 02:55:18,400 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:55:18,400 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9d5b411b5fe67e5bb007f4d36e3e11e5, store=B 2023-04-25 02:55:18,400 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:55:18,400 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9d5b411b5fe67e5bb007f4d36e3e11e5, store=C 2023-04-25 02:55:18,400 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:55:18,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(9158): Flush requested on 9d5b411b5fe67e5bb007f4d36e3e11e5 2023-04-25 02:55:18,407 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. as already flushing 2023-04-25 02:55:18,435 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:18,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54924 deadline: 1682391378427, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:18,439 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:18,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 220 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54884 deadline: 1682391378433, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:18,537 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:18,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54924 deadline: 1682391378536, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:18,543 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:18,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 222 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54884 deadline: 1682391378540, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:18,745 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:18,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54924 deadline: 1682391378738, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:18,746 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:18,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 224 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54884 deadline: 1682391378745, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:18,814 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304259282819193c44462b0130d7f74931c0f to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304259282819193c44462b0130d7f74931c0f 2023-04-25 02:55:18,819 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=474, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/.tmp/A/090c421124fe44b98ff74ae2318d0481 2023-04-25 02:55:18,835 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=474 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/.tmp/B/7e0bb30ba5ae4d09b3b44e47268e1278 2023-04-25 02:55:18,849 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=474 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/.tmp/C/67b5bda6cb9b4513ac61fd00f7b73765 2023-04-25 02:55:18,855 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/.tmp/A/090c421124fe44b98ff74ae2318d0481 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/A/090c421124fe44b98ff74ae2318d0481 2023-04-25 02:55:18,858 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/A/090c421124fe44b98ff74ae2318d0481, entries=150, sequenceid=474, filesize=25.4 K 2023-04-25 02:55:18,859 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/.tmp/B/7e0bb30ba5ae4d09b3b44e47268e1278 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/B/7e0bb30ba5ae4d09b3b44e47268e1278 2023-04-25 02:55:18,863 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/B/7e0bb30ba5ae4d09b3b44e47268e1278, entries=150, sequenceid=474, filesize=11.9 K 2023-04-25 02:55:18,863 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/.tmp/C/67b5bda6cb9b4513ac61fd00f7b73765 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/C/67b5bda6cb9b4513ac61fd00f7b73765 2023-04-25 02:55:18,867 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/C/67b5bda6cb9b4513ac61fd00f7b73765, entries=150, sequenceid=474, filesize=11.9 K 2023-04-25 02:55:18,868 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 9d5b411b5fe67e5bb007f4d36e3e11e5 in 468ms, sequenceid=474, compaction requested=true 2023-04-25 02:55:18,868 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9d5b411b5fe67e5bb007f4d36e3e11e5: 2023-04-25 02:55:18,868 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. 2023-04-25 02:55:18,868 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:55:18,868 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:55:18,868 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:18,868 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:55:18,868 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:55:18,869 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:18,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:18,870 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:18,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:18,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:18,870 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:18,870 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:55:18,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:18,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:18,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:18,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:18,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:18,872 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:55:18,872 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5c87705c[Count = 0] remaining members to acquire global barrier 2023-04-25 02:55:18,872 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:55:18,872 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:18,874 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:18,874 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:18,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:18,874 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:55:18,875 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:55:18,875 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:55:18,876 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:55:18,876 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:18,876 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:18,876 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:55:18,876 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:55:18,876 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:55:18,876 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:55:18,876 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:55:18,876 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:55:18,876 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:55:18,876 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:55:18,878 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:18,878 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:18,878 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:18,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:18,878 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:18,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:18,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:18,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:18,878 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:18,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:18,879 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:18,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:18,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:18,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:18,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:18,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:18,880 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:18,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:18,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:18,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:18,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:18,882 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:18,882 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:18,882 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:18,882 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:18,882 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:18,882 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:18,882 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:18,882 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:18,882 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:18,882 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:18,882 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:55:18,882 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:55:18,882 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:18,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:18,883 DEBUG [Thread-2260] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:55:18,883 DEBUG [Thread-2260] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:55:18,883 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:18,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:18,884 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:55:18,983 DEBUG [Thread-2260] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:55:18,983 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:55:18,985 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:55:18,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:55:18,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:55:18,986 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:55:18,986 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:18,986 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:55:18,986 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:55:18,987 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:18,987 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:18,989 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:18,989 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:18,989 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:18,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:18,989 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:18,989 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:55:18,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:18,990 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:18,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:55:18,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:18,990 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:55:18,990 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:55:18,990 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:55:18,990 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:55:18,990 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:18,990 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:55:18,990 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:55:18,991 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:55:18,991 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. 2023-04-25 02:55:18,991 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. started... 2023-04-25 02:55:18,991 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 9d5b411b5fe67e5bb007f4d36e3e11e5 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-25 02:55:18,991 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9d5b411b5fe67e5bb007f4d36e3e11e5, store=A 2023-04-25 02:55:18,991 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:55:18,991 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9d5b411b5fe67e5bb007f4d36e3e11e5, store=B 2023-04-25 02:55:18,991 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:55:18,991 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9d5b411b5fe67e5bb007f4d36e3e11e5, store=C 2023-04-25 02:55:18,991 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:55:19,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(9158): Flush requested on 9d5b411b5fe67e5bb007f4d36e3e11e5 2023-04-25 02:55:19,053 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. as already flushing 2023-04-25 02:55:19,113 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:19,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 234 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54884 deadline: 1682391379105, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:19,117 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:19,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 225 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54924 deadline: 1682391379107, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:19,218 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:19,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 236 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54884 deadline: 1682391379214, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:19,223 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:19,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54924 deadline: 1682391379218, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:19,404 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230425be9013c86466429c924b2088c228e97c to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425be9013c86466429c924b2088c228e97c 2023-04-25 02:55:19,422 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:19,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 238 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54884 deadline: 1682391379420, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:19,429 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:19,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 229 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54924 deadline: 1682391379425, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:19,729 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:19,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 240 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54884 deadline: 1682391379724, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:19,737 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:19,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 231 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54924 deadline: 1682391379730, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:19,809 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=486, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/.tmp/A/140f06ca977e4666814670510e7c6fa2 2023-04-25 02:55:20,217 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:20,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54940 deadline: 1682391380213, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:20,219 DEBUG [Thread-2254] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8162 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5., hostname=jenkins-hbase4.apache.org,46697,1682391220023, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor28.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-25 02:55:20,220 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=486 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/.tmp/B/b00219107a614a578ce1ff9ef9b51c3f 2023-04-25 02:55:20,227 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:20,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54886 deadline: 1682391380222, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:20,229 DEBUG [Thread-2258] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8174 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5., hostname=jenkins-hbase4.apache.org,46697,1682391220023, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor28.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-25 02:55:20,230 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=486 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/.tmp/C/ed72f11460d34a31b8dae931145a746a 2023-04-25 02:55:20,235 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:20,235 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/.tmp/A/140f06ca977e4666814670510e7c6fa2 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/A/140f06ca977e4666814670510e7c6fa2 2023-04-25 02:55:20,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 242 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54884 deadline: 1682391380230, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:20,239 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/A/140f06ca977e4666814670510e7c6fa2, entries=150, sequenceid=486, filesize=25.4 K 2023-04-25 02:55:20,240 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/.tmp/B/b00219107a614a578ce1ff9ef9b51c3f as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/B/b00219107a614a578ce1ff9ef9b51c3f 2023-04-25 02:55:20,243 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/B/b00219107a614a578ce1ff9ef9b51c3f, entries=150, sequenceid=486, filesize=11.9 K 2023-04-25 02:55:20,243 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/.tmp/C/ed72f11460d34a31b8dae931145a746a as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/C/ed72f11460d34a31b8dae931145a746a 2023-04-25 02:55:20,245 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:20,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54894 deadline: 1682391380238, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:20,245 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:20,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 233 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54924 deadline: 1682391380242, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:20,247 DEBUG [Thread-2250] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8188 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5., hostname=jenkins-hbase4.apache.org,46697,1682391220023, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor28.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-25 02:55:20,249 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/C/ed72f11460d34a31b8dae931145a746a, entries=150, sequenceid=486, filesize=11.9 K 2023-04-25 02:55:20,250 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 9d5b411b5fe67e5bb007f4d36e3e11e5 in 1259ms, sequenceid=486, compaction requested=true 2023-04-25 02:55:20,250 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9d5b411b5fe67e5bb007f4d36e3e11e5: 2023-04-25 02:55:20,250 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. 2023-04-25 02:55:20,250 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:55:20,250 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:55:20,250 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:20,250 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:55:20,250 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:55:20,252 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:20,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:20,252 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:20,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:20,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:20,252 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:20,252 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:55:20,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:20,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:20,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:20,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:20,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:20,253 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:55:20,253 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@60e3f810[Count = 0] remaining members to acquire global barrier 2023-04-25 02:55:20,253 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:55:20,253 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:20,255 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:20,255 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:20,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:20,255 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:55:20,255 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:55:20,255 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:55:20,255 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:20,255 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:55:20,256 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:20,256 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:55:20,256 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:20,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:20,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:20,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:20,256 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:55:20,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:20,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:20,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:20,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:20,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:20,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:20,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:20,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:55:20,258 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:55:20,258 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:55:20,258 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:55:20,258 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:55:20,258 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:55:20,258 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:55:20,261 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:20,261 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:20,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:20,261 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:20,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:20,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:20,261 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:20,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:20,261 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:20,261 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:20,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:20,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:20,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:20,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:20,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:20,262 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:20,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:20,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:20,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:20,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:20,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:20,264 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:20,264 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:20,264 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:20,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:20,264 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:20,264 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:20,264 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:20,264 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:20,264 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:20,264 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:20,265 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:20,265 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:20,264 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:55:20,265 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:55:20,265 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:20,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:20,265 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:55:20,265 DEBUG [Thread-2260] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:55:20,265 DEBUG [Thread-2260] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:55:20,365 DEBUG [Thread-2260] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:55:20,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:55:20,367 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:55:20,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:55:20,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:55:20,368 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:55:20,368 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:20,368 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:55:20,368 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:55:20,369 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:20,369 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:20,370 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:20,370 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:20,370 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:20,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:20,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:20,370 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:20,370 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:55:20,371 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:20,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:55:20,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:20,371 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:55:20,371 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:55:20,371 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:55:20,371 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:55:20,371 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:20,371 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:55:20,372 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:55:20,372 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:55:20,372 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. 2023-04-25 02:55:20,372 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. started... 2023-04-25 02:55:20,372 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 9d5b411b5fe67e5bb007f4d36e3e11e5 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-25 02:55:20,372 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9d5b411b5fe67e5bb007f4d36e3e11e5, store=A 2023-04-25 02:55:20,372 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:55:20,372 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9d5b411b5fe67e5bb007f4d36e3e11e5, store=B 2023-04-25 02:55:20,372 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:55:20,372 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9d5b411b5fe67e5bb007f4d36e3e11e5, store=C 2023-04-25 02:55:20,372 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:55:20,386 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230425e2dfced89dbc43cb87fc6f5c7bcb7643 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425e2dfced89dbc43cb87fc6f5c7bcb7643 2023-04-25 02:55:20,791 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=506, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/.tmp/A/a3a34b8e3d4941d1977b128b8d773072 2023-04-25 02:55:21,201 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=506 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/.tmp/B/66a4d84d12e24db0abb6101e481f225e 2023-04-25 02:55:21,212 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=506 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/.tmp/C/123c0380fc2e44919ac8b3efc9a443bf 2023-04-25 02:55:21,215 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/.tmp/A/a3a34b8e3d4941d1977b128b8d773072 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/A/a3a34b8e3d4941d1977b128b8d773072 2023-04-25 02:55:21,220 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/A/a3a34b8e3d4941d1977b128b8d773072, entries=150, sequenceid=506, filesize=25.4 K 2023-04-25 02:55:21,220 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/.tmp/B/66a4d84d12e24db0abb6101e481f225e as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/B/66a4d84d12e24db0abb6101e481f225e 2023-04-25 02:55:21,224 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/B/66a4d84d12e24db0abb6101e481f225e, entries=150, sequenceid=506, filesize=11.9 K 2023-04-25 02:55:21,225 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/.tmp/C/123c0380fc2e44919ac8b3efc9a443bf as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/C/123c0380fc2e44919ac8b3efc9a443bf 2023-04-25 02:55:21,232 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/C/123c0380fc2e44919ac8b3efc9a443bf, entries=150, sequenceid=506, filesize=11.9 K 2023-04-25 02:55:21,233 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=0 B/0 for 9d5b411b5fe67e5bb007f4d36e3e11e5 in 861ms, sequenceid=506, compaction requested=true 2023-04-25 02:55:21,233 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9d5b411b5fe67e5bb007f4d36e3e11e5: 2023-04-25 02:55:21,233 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. 2023-04-25 02:55:21,233 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:55:21,233 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:55:21,233 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:21,233 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:55:21,233 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:55:21,235 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:21,235 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:21,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:21,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:21,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:21,235 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:21,235 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:55:21,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:21,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:21,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:21,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:21,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:21,238 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:55:21,238 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2312bb39[Count = 0] remaining members to acquire global barrier 2023-04-25 02:55:21,238 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:55:21,238 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:21,240 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:21,240 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:21,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:21,241 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:55:21,241 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:55:21,241 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:21,241 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:55:21,241 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:55:21,242 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:21,242 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:55:21,242 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:21,242 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:55:21,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:21,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:21,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:21,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:21,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:21,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:21,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:21,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:21,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:21,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:21,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:55:21,244 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:55:21,244 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:55:21,244 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:55:21,244 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:55:21,244 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:55:21,244 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:55:21,246 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:21,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:21,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:21,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:21,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:21,246 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:21,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:21,247 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:21,247 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:21,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:21,247 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:21,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:21,247 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:21,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:21,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:21,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:21,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:21,248 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:21,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:21,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:21,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:21,255 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:21,255 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:21,255 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:21,255 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:21,255 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:55:21,255 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:21,255 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:55:21,255 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:21,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:21,256 DEBUG [Thread-2260] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:55:21,256 DEBUG [Thread-2260] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:55:21,256 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:21,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:21,255 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:21,256 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:21,256 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:21,256 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:21,256 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:21,356 DEBUG [Thread-2260] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:55:21,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:55:21,358 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:55:21,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:55:21,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:55:21,360 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:55:21,360 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:21,360 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:55:21,360 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:55:21,361 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:21,361 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:21,362 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:21,362 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:21,362 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:21,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:21,362 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:21,362 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:55:21,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:21,362 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:21,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:55:21,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:21,363 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:55:21,363 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:55:21,363 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:55:21,363 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:55:21,363 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:21,363 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:55:21,364 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:55:21,364 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:55:21,364 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. 2023-04-25 02:55:21,364 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. started... 2023-04-25 02:55:21,364 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 9d5b411b5fe67e5bb007f4d36e3e11e5 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-25 02:55:21,364 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9d5b411b5fe67e5bb007f4d36e3e11e5, store=A 2023-04-25 02:55:21,364 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:55:21,364 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9d5b411b5fe67e5bb007f4d36e3e11e5, store=B 2023-04-25 02:55:21,364 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:55:21,364 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9d5b411b5fe67e5bb007f4d36e3e11e5, store=C 2023-04-25 02:55:21,364 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:55:21,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(9158): Flush requested on 9d5b411b5fe67e5bb007f4d36e3e11e5 2023-04-25 02:55:21,450 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. as already flushing 2023-04-25 02:55:21,512 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682391321479, max timestamp=1682391321510Num uniques 859; 2023-04-25 02:55:21,512 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682391321479, max timestamp=1682391321510Num uniques 859; 2023-04-25 02:55:21,512 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-25 02:55:21,512 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-25 02:55:21,512 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682391321479, max timestamp=1682391321510Num uniques 859; 2023-04-25 02:55:21,513 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-25 02:55:21,615 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1682391321558, max timestamp=1682391321612Num uniques 1068; 2023-04-25 02:55:21,615 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=7688, suffix heap size=139040,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=168 2023-04-25 02:55:21,615 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1682391321558, max timestamp=1682391321612Num uniques 1068; 2023-04-25 02:55:21,615 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=7688, suffix heap size=139040,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=168 2023-04-25 02:55:21,615 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1682391321558, max timestamp=1682391321612Num uniques 1068; 2023-04-25 02:55:21,615 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=7688, suffix heap size=139040,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=168 2023-04-25 02:55:21,705 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1682391321674, max timestamp=1682391321703Num uniques 1027; 2023-04-25 02:55:21,705 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=10392, suffix heap size=134776,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=227 2023-04-25 02:55:21,706 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1682391321674, max timestamp=1682391321703Num uniques 1027; 2023-04-25 02:55:21,706 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1682391321674, max timestamp=1682391321703Num uniques 1027; 2023-04-25 02:55:21,706 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=10392, suffix heap size=134776,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=227 2023-04-25 02:55:21,706 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=10392, suffix heap size=134776,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=227 2023-04-25 02:55:21,783 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1682391321760, max timestamp=1682391321781Num uniques 1086; 2023-04-25 02:55:21,783 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1682391321760, max timestamp=1682391321781Num uniques 1086; 2023-04-25 02:55:21,783 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1682391321760, max timestamp=1682391321781Num uniques 1086; 2023-04-25 02:55:21,784 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=10806, suffix heap size=140912,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=236 2023-04-25 02:55:21,784 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=10806, suffix heap size=140912,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=236 2023-04-25 02:55:21,784 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=10806, suffix heap size=140912,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=236 2023-04-25 02:55:21,786 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023042567177bf144144e7c87d64c73089449eb to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042567177bf144144e7c87d64c73089449eb 2023-04-25 02:55:21,791 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=513, memsize=8.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/.tmp/A/f3c5113141344f78a252d9fe29299a30 2023-04-25 02:55:21,808 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=513 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/.tmp/B/102dde04601f4b6491d366c3895677e0 2023-04-25 02:55:21,906 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1682391321870, max timestamp=1682391321903Num uniques 1095; 2023-04-25 02:55:21,906 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=8930, suffix heap size=141848,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=195 2023-04-25 02:55:21,906 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1682391321870, max timestamp=1682391321903Num uniques 1095; 2023-04-25 02:55:21,906 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1682391321870, max timestamp=1682391321903Num uniques 1095; 2023-04-25 02:55:21,906 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=8930, suffix heap size=141848,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=195 2023-04-25 02:55:21,906 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=8930, suffix heap size=141848,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=195 2023-04-25 02:55:22,018 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1682391321994, max timestamp=1682391322016Num uniques 1054; 2023-04-25 02:55:22,018 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=7050, suffix heap size=137584,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=154 2023-04-25 02:55:22,019 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1682391321994, max timestamp=1682391322016Num uniques 1054; 2023-04-25 02:55:22,019 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=7050, suffix heap size=137584,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=154 2023-04-25 02:55:22,019 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1682391321994, max timestamp=1682391322016Num uniques 1054; 2023-04-25 02:55:22,019 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=7050, suffix heap size=137584,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=154 2023-04-25 02:55:22,085 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1682391322059, max timestamp=1682391322083Num uniques 1013; 2023-04-25 02:55:22,085 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=7458, suffix heap size=133320,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=163 2023-04-25 02:55:22,085 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1682391322059, max timestamp=1682391322083Num uniques 1013; 2023-04-25 02:55:22,085 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1682391322059, max timestamp=1682391322083Num uniques 1013; 2023-04-25 02:55:22,086 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=7458, suffix heap size=133320,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=163 2023-04-25 02:55:22,086 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=7458, suffix heap size=133320,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=163 2023-04-25 02:55:22,165 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7872, totalHeapSize=18128, min timestamp=1682391322127, max timestamp=1682391322163Num uniques 1022; 2023-04-25 02:55:22,165 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=7872, suffix heap size=134256,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=172 2023-04-25 02:55:22,165 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7872, totalHeapSize=18128, min timestamp=1682391322127, max timestamp=1682391322163Num uniques 1022; 2023-04-25 02:55:22,165 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7872, totalHeapSize=18128, min timestamp=1682391322127, max timestamp=1682391322163Num uniques 1022; 2023-04-25 02:55:22,166 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=7872, suffix heap size=134256,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=172 2023-04-25 02:55:22,165 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=7872, suffix heap size=134256,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=172 2023-04-25 02:55:22,234 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=513 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/.tmp/C/616aca292ee64a4cb7f7bcce4a25a9e3 2023-04-25 02:55:22,238 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/.tmp/A/f3c5113141344f78a252d9fe29299a30 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/A/f3c5113141344f78a252d9fe29299a30 2023-04-25 02:55:22,248 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/A/f3c5113141344f78a252d9fe29299a30, entries=150, sequenceid=513, filesize=25.4 K 2023-04-25 02:55:22,250 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/.tmp/B/102dde04601f4b6491d366c3895677e0 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/B/102dde04601f4b6491d366c3895677e0 2023-04-25 02:55:22,254 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/B/102dde04601f4b6491d366c3895677e0, entries=100, sequenceid=513, filesize=9.5 K 2023-04-25 02:55:22,255 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/.tmp/C/616aca292ee64a4cb7f7bcce4a25a9e3 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/C/616aca292ee64a4cb7f7bcce4a25a9e3 2023-04-25 02:55:22,258 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/C/616aca292ee64a4cb7f7bcce4a25a9e3, entries=100, sequenceid=513, filesize=9.5 K 2023-04-25 02:55:22,258 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=134.18 KB/137400 for 9d5b411b5fe67e5bb007f4d36e3e11e5 in 894ms, sequenceid=513, compaction requested=true 2023-04-25 02:55:22,259 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9d5b411b5fe67e5bb007f4d36e3e11e5: 2023-04-25 02:55:22,259 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. 2023-04-25 02:55:22,259 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:55:22,259 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:55:22,259 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:22,259 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:55:22,259 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:55:22,260 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:22,260 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:22,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:22,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:22,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:22,261 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:22,261 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:55:22,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:22,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:22,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:22,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:22,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:22,263 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:55:22,263 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@396ac5d0[Count = 0] remaining members to acquire global barrier 2023-04-25 02:55:22,263 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:55:22,263 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:22,264 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:22,264 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:22,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:22,264 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:55:22,264 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:55:22,264 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:55:22,264 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:22,264 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:55:22,267 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:55:22,267 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:22,267 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:22,267 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:55:22,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:22,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:22,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:22,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:22,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:22,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:22,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:22,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:22,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(9158): Flush requested on 9d5b411b5fe67e5bb007f4d36e3e11e5 2023-04-25 02:55:22,269 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 9d5b411b5fe67e5bb007f4d36e3e11e5 3/3 column families, dataSize=140.89 KB heapSize=404.39 KB 2023-04-25 02:55:22,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:22,269 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9d5b411b5fe67e5bb007f4d36e3e11e5, store=A 2023-04-25 02:55:22,269 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-25 02:55:22,269 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9d5b411b5fe67e5bb007f4d36e3e11e5, store=B 2023-04-25 02:55:22,269 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-25 02:55:22,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:22,269 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9d5b411b5fe67e5bb007f4d36e3e11e5, store=C 2023-04-25 02:55:22,269 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-25 02:55:22,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:55:22,270 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:55:22,270 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:55:22,270 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:55:22,270 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:55:22,270 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:55:22,270 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:55:22,273 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:22,273 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:22,273 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:22,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:22,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:22,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:22,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:22,273 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:22,274 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:22,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:22,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:22,274 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:22,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:22,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:22,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:22,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:22,275 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:22,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:22,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:22,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:22,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:22,277 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:22,277 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:22,277 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:22,277 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:22,277 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:22,277 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:22,277 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:22,277 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:22,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:22,277 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:22,277 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:22,277 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:55:22,278 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:55:22,278 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:55:22,277 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:22,278 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:22,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:22,278 DEBUG [Thread-2260] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:55:22,278 DEBUG [Thread-2260] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:55:22,306 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:22,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 318 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54924 deadline: 1682391382293, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:22,318 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:22,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 326 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54884 deadline: 1682391382306, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:22,378 DEBUG [Thread-2260] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:55:22,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:55:22,380 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:55:22,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:55:22,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:55:22,388 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:55:22,388 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:22,388 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:55:22,388 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:55:22,389 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:22,389 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:22,390 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:22,390 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:22,390 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:22,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:22,390 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:22,390 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:55:22,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:22,391 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:22,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:55:22,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:22,391 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:55:22,392 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:55:22,392 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:55:22,392 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:55:22,392 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:22,392 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:55:22,395 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:55:22,395 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:55:22,395 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. 2023-04-25 02:55:22,395 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. started... 2023-04-25 02:55:22,395 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. as already flushing 2023-04-25 02:55:22,395 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. 2023-04-25 02:55:22,396 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:55:22,396 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:55:22,396 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:22,396 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:55:22,396 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:55:22,398 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:22,398 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:22,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:22,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:22,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:22,399 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:22,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:22,399 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:55:22,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:22,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:22,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:22,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:22,401 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:55:22,401 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5e085731[Count = 0] remaining members to acquire global barrier 2023-04-25 02:55:22,401 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:55:22,401 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:22,402 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:22,402 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:22,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:22,403 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:55:22,403 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:55:22,403 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:55:22,406 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:55:22,406 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:22,406 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:55:22,406 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:22,407 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:55:22,407 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:55:22,407 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:55:22,407 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:55:22,407 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:55:22,407 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:55:22,407 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:55:22,410 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:22,410 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:22,410 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:22,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:22,411 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:22,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:22,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:22,412 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:22,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:22,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:22,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:22,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:22,412 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:22,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:22,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:22,413 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:22,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 320 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54924 deadline: 1682391382408, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:22,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:22,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:22,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:22,416 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:22,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:22,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:22,420 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:22,420 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:22,420 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:22,420 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:22,420 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:55:22,420 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:55:22,420 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:22,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:22,420 DEBUG [Thread-2260] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:55:22,420 DEBUG [Thread-2260] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:55:22,420 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:22,420 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:22,420 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:22,420 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:22,420 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:22,420 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:22,421 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:22,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:22,426 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:22,426 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 328 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54884 deadline: 1682391382419, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:22,520 DEBUG [Thread-2260] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:55:22,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:55:22,522 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:55:22,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:55:22,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:55:22,524 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:55:22,524 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:22,524 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:55:22,524 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:55:22,524 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:22,524 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:22,525 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:22,525 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:22,526 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:22,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:22,526 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:22,526 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:55:22,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:22,526 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:22,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:55:22,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:22,527 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:55:22,527 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:55:22,527 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:55:22,527 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:55:22,527 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:22,527 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:55:22,527 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:55:22,527 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:55:22,527 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. 2023-04-25 02:55:22,527 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. started... 2023-04-25 02:55:22,527 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. as already flushing 2023-04-25 02:55:22,527 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. 2023-04-25 02:55:22,528 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:55:22,528 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:55:22,528 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:22,528 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:55:22,528 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:55:22,529 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:22,529 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:22,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:22,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:22,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:22,529 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:22,530 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:55:22,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:22,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:22,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:22,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:22,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:22,531 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:55:22,531 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@21d40798[Count = 0] remaining members to acquire global barrier 2023-04-25 02:55:22,531 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:55:22,531 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:22,532 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:22,532 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:22,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:22,532 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:55:22,532 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:22,532 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:55:22,532 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:55:22,532 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:55:22,534 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:55:22,534 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:22,534 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:22,534 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:55:22,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:22,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:22,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:22,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:22,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:22,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:22,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:22,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:22,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:22,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:22,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:55:22,540 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:55:22,540 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:55:22,540 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:55:22,540 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:55:22,540 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:55:22,540 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:55:22,541 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:22,542 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:22,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:22,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:22,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:22,542 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:22,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:22,542 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:22,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:22,542 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:22,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:22,542 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:22,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:22,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:22,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:22,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:22,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:22,543 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:22,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:22,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:22,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:22,548 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:22,548 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:22,548 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:22,548 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:22,548 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:22,548 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:22,548 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:22,548 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:22,548 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:55:22,549 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:55:22,548 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:22,549 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:22,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:22,549 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:22,549 DEBUG [Thread-2260] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:55:22,549 DEBUG [Thread-2260] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:55:22,549 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:22,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:22,617 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:22,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 322 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54924 deadline: 1682391382615, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:22,637 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:22,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 330 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54884 deadline: 1682391382627, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:22,649 DEBUG [Thread-2260] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:55:22,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:55:22,653 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:55:22,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:55:22,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:55:22,655 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:55:22,655 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:22,655 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:55:22,655 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:55:22,656 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:22,656 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:22,659 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:22,659 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:22,659 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:22,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:22,660 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:22,660 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:55:22,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:22,660 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:22,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:55:22,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:22,660 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:55:22,661 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:55:22,661 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:55:22,661 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:55:22,661 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:22,661 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:55:22,661 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:55:22,661 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:55:22,661 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. 2023-04-25 02:55:22,661 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. started... 2023-04-25 02:55:22,661 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. as already flushing 2023-04-25 02:55:22,661 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. 2023-04-25 02:55:22,662 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:55:22,662 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:55:22,662 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:22,662 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:55:22,662 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:55:22,663 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:22,663 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:22,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:22,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:22,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:22,664 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:22,664 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:55:22,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:22,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:22,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:22,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:22,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:22,672 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:55:22,672 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1729abe8[Count = 0] remaining members to acquire global barrier 2023-04-25 02:55:22,672 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:55:22,672 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:22,673 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:22,673 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:22,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:22,673 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:55:22,673 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:55:22,673 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:55:22,673 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:22,673 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:55:22,675 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:55:22,675 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:22,675 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:22,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:22,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:22,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:22,675 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:55:22,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:22,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:22,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:22,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:22,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:22,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:22,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:22,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:55:22,679 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:55:22,679 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:55:22,679 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:55:22,679 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:55:22,679 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:55:22,679 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:55:22,681 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:22,681 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:22,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:22,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:22,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:22,681 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:22,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:22,681 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:22,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:22,682 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:22,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:22,682 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:22,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:22,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:22,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:22,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:22,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:22,683 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:22,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:22,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:22,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:22,685 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:22,685 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:22,685 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:22,685 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:22,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:22,685 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:22,685 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:22,685 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:55:22,685 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:55:22,685 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:22,685 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:22,685 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:22,685 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:22,685 DEBUG [Thread-2260] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:55:22,685 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:22,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:22,685 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:22,685 DEBUG [Thread-2260] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:55:22,704 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304254255a15a98ea4b90a77d4646a65cb9db to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304254255a15a98ea4b90a77d4646a65cb9db 2023-04-25 02:55:22,786 DEBUG [Thread-2260] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:55:22,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:55:22,788 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:55:22,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:55:22,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:55:22,790 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:55:22,790 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:22,790 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:55:22,790 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:55:22,790 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:22,790 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:22,792 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:22,792 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:22,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:22,792 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:22,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:22,792 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:22,793 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:55:22,793 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:22,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:55:22,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:22,793 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:55:22,793 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:55:22,793 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:55:22,794 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:55:22,794 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:22,794 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:55:22,794 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:55:22,794 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:55:22,794 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. 2023-04-25 02:55:22,794 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. started... 2023-04-25 02:55:22,794 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. as already flushing 2023-04-25 02:55:22,794 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. 2023-04-25 02:55:22,794 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:55:22,794 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:55:22,794 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:22,794 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:55:22,794 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:55:22,796 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:22,796 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:22,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:22,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:22,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:22,797 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:22,797 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:55:22,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:22,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:22,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:22,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:22,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:22,799 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:55:22,799 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7802b898[Count = 0] remaining members to acquire global barrier 2023-04-25 02:55:22,799 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:55:22,799 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:22,802 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:22,802 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:22,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:22,802 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:55:22,802 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:55:22,802 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:55:22,803 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:55:22,803 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:22,803 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:22,803 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:55:22,804 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:55:22,804 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:55:22,804 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:55:22,804 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:55:22,804 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:55:22,804 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:55:22,804 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:55:22,805 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:22,805 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:22,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:22,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:22,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:22,805 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:22,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:22,805 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:22,806 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:22,806 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:22,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:22,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:22,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:22,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:22,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:22,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:22,807 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:22,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:22,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:22,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:22,809 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:22,809 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:22,809 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:22,809 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:22,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:22,809 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:22,809 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:55:22,809 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:55:22,809 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:22,809 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:22,810 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:22,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:22,810 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:55:22,810 DEBUG [Thread-2260] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:55:22,810 DEBUG [Thread-2260] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:55:22,810 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:22,810 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:22,810 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:22,810 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:22,910 DEBUG [Thread-2260] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:55:22,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:55:22,912 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:55:22,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:55:22,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:55:22,913 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:55:22,913 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:22,913 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:55:22,913 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:55:22,913 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:22,914 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:22,916 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:22,916 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:22,916 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:22,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:22,916 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:22,916 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:55:22,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:22,917 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:22,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:55:22,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:22,917 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:55:22,917 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:55:22,917 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:55:22,918 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:55:22,918 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:22,918 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:55:22,918 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:55:22,918 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:55:22,918 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. 2023-04-25 02:55:22,918 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. started... 2023-04-25 02:55:22,918 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. as already flushing 2023-04-25 02:55:22,918 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. 2023-04-25 02:55:22,918 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:55:22,918 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:55:22,918 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:22,918 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:55:22,918 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:55:22,920 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:22,920 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:22,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:22,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:22,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:22,920 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:22,920 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:55:22,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:22,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:22,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:22,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:22,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:22,921 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:55:22,922 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@64afc976[Count = 0] remaining members to acquire global barrier 2023-04-25 02:55:22,922 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:55:22,922 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:22,923 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:22,923 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:22,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:22,923 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:55:22,923 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:55:22,923 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:55:22,924 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:55:22,924 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:22,924 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:22,924 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:55:22,924 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:55:22,924 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:55:22,924 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:55:22,924 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:55:22,925 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:55:22,925 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:55:22,925 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:55:22,926 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:22,926 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:22,927 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:22,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:22,927 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:22,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:22,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:22,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:22,927 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:22,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:22,927 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:22,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:22,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:22,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:22,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:22,928 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:22,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:22,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:22,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:22,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:22,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:22,930 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:22,930 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:22,930 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:22,930 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:22,930 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:55:22,930 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:55:22,930 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:22,930 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:22,930 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:22,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:22,930 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:55:22,930 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:22,930 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:22,931 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:22,930 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:22,931 DEBUG [Thread-2260] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:55:22,931 DEBUG [Thread-2260] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:55:22,931 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:22,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:22,932 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:22,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 324 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54924 deadline: 1682391382919, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:22,951 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:22,951 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 332 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54884 deadline: 1682391382944, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:23,031 DEBUG [Thread-2260] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:55:23,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:55:23,034 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:55:23,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:55:23,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:55:23,038 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:55:23,038 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:23,038 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:55:23,038 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:55:23,038 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:23,038 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:23,039 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:23,039 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:23,039 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:23,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:23,040 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:23,040 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:55:23,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:23,040 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:23,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:55:23,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:23,041 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:55:23,044 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:55:23,044 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:55:23,044 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:55:23,044 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:23,044 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:55:23,044 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:55:23,044 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. 2023-04-25 02:55:23,044 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. started... 2023-04-25 02:55:23,044 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:55:23,044 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. as already flushing 2023-04-25 02:55:23,044 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. 2023-04-25 02:55:23,044 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:55:23,044 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:55:23,044 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:23,044 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:55:23,044 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:55:23,055 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:23,055 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:23,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:23,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:23,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:23,055 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:23,055 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:55:23,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:23,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:23,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:23,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:23,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:23,061 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:55:23,061 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6546fabc[Count = 0] remaining members to acquire global barrier 2023-04-25 02:55:23,061 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:55:23,061 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:23,062 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:23,062 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:23,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:23,063 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:55:23,063 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:55:23,063 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:55:23,063 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:23,063 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:55:23,064 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:55:23,064 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:23,064 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:23,064 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:55:23,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:23,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:23,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:23,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:23,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:23,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:23,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:23,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:23,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:23,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:23,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:55:23,067 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:55:23,067 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:55:23,067 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:55:23,067 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:55:23,067 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:55:23,067 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:55:23,068 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:23,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:23,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:23,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:23,069 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:23,069 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:23,069 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:23,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:23,069 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:23,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:23,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:23,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:23,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:23,076 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:23,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:23,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:23,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:23,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:23,077 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:23,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:23,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:23,080 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:23,080 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:23,080 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:23,080 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:55:23,080 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:55:23,080 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:23,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:23,080 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:23,080 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:23,080 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:23,080 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:23,080 DEBUG [Thread-2260] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:55:23,080 DEBUG [Thread-2260] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:55:23,080 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:23,081 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:23,081 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:23,081 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:23,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:23,110 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=671, memsize=47.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/.tmp/A/d31fd5cb265748fda29a79ca2593bc32 2023-04-25 02:55:23,181 DEBUG [Thread-2260] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:55:23,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:55:23,183 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:55:23,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:55:23,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:55:23,184 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:55:23,184 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:23,184 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:55:23,184 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:55:23,184 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:23,184 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:23,185 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:23,185 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:23,186 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:23,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:23,186 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:23,186 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:55:23,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:23,186 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:23,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:55:23,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:23,187 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:55:23,187 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:55:23,187 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:55:23,187 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:55:23,187 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:23,187 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:55:23,187 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:55:23,187 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:55:23,187 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. 2023-04-25 02:55:23,187 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. started... 2023-04-25 02:55:23,187 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. as already flushing 2023-04-25 02:55:23,187 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. 2023-04-25 02:55:23,187 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:55:23,187 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:55:23,187 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:23,187 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:55:23,187 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:55:23,189 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:23,189 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:23,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:23,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:23,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:23,189 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:23,189 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:55:23,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:23,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:23,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:23,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:23,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:23,190 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:55:23,190 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:55:23,190 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3fea7b17[Count = 0] remaining members to acquire global barrier 2023-04-25 02:55:23,191 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:23,192 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:23,192 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:23,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:23,192 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:55:23,192 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:55:23,192 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:55:23,192 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:23,192 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:55:23,194 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:23,194 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:55:23,194 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:23,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:23,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:23,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:23,194 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:55:23,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:23,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:23,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:23,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:23,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:23,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:23,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:23,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:55:23,196 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:55:23,196 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:55:23,196 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:55:23,196 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:55:23,196 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:55:23,196 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:55:23,197 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:23,197 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:23,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:23,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:23,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:23,198 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:23,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:23,198 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:23,198 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:23,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:23,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:23,198 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:23,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:23,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:23,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:23,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:23,199 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:23,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:23,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:23,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:23,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:23,201 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:23,201 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:23,201 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:23,202 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:23,202 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:23,202 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:23,202 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:23,202 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:55:23,202 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:55:23,202 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:23,202 DEBUG [Thread-2260] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:55:23,202 DEBUG [Thread-2260] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:55:23,202 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:55:23,202 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:23,202 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:23,202 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:23,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:23,203 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:23,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:23,302 DEBUG [Thread-2260] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:55:23,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:55:23,305 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:55:23,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:55:23,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:55:23,307 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:55:23,307 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:23,307 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:55:23,307 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:55:23,308 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:23,308 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:23,309 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:23,309 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:23,309 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:23,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:23,309 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:23,309 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:55:23,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:23,310 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:23,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:55:23,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:23,310 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:55:23,310 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:55:23,310 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:55:23,310 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:55:23,310 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:23,310 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:55:23,311 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:55:23,311 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:55:23,311 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. 2023-04-25 02:55:23,311 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. started... 2023-04-25 02:55:23,311 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. as already flushing 2023-04-25 02:55:23,311 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. 2023-04-25 02:55:23,311 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:55:23,311 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:55:23,311 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:23,311 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:55:23,311 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:55:23,313 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:23,313 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:23,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:23,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:23,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:23,314 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:23,314 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:55:23,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:23,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:23,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:23,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:23,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:23,315 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:55:23,315 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:55:23,315 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@52c1c20a[Count = 0] remaining members to acquire global barrier 2023-04-25 02:55:23,315 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:23,316 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:23,316 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:23,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:23,316 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:55:23,316 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:55:23,316 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:55:23,316 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:23,316 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:55:23,317 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:55:23,317 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:23,317 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:23,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:23,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:23,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:23,318 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:55:23,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:23,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:23,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:23,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:23,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:23,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:23,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:23,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:55:23,320 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:55:23,320 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:55:23,320 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:55:23,320 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:55:23,320 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:55:23,320 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:55:23,321 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:23,321 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:23,321 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:23,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:23,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:23,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:23,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:23,321 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:23,322 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:23,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:23,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:23,322 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:23,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:23,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:23,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:23,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:23,323 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:23,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:23,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:23,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:23,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:23,326 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:23,326 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:23,326 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:23,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:23,326 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:23,327 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:23,327 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:55:23,327 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:55:23,326 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:23,327 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:23,327 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:23,327 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:23,327 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:23,327 DEBUG [Thread-2260] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:55:23,327 DEBUG [Thread-2260] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:55:23,327 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:23,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:23,327 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:23,327 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:55:23,427 DEBUG [Thread-2260] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:55:23,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:55:23,430 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:55:23,431 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:55:23,431 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:55:23,431 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:55:23,431 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:23,431 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:55:23,431 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:55:23,432 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:23,432 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:23,433 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:23,433 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:23,433 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:23,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:23,433 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:23,433 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:55:23,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:23,434 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:23,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:55:23,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:23,434 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:55:23,434 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:55:23,434 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:55:23,434 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:55:23,434 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:23,434 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:55:23,434 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:55:23,434 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:55:23,435 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. 2023-04-25 02:55:23,435 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. started... 2023-04-25 02:55:23,435 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. as already flushing 2023-04-25 02:55:23,435 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. 2023-04-25 02:55:23,435 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:55:23,435 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:55:23,435 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:23,435 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:55:23,435 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:55:23,437 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:23,437 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:23,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:23,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:23,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:23,437 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:23,437 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:55:23,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:23,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:23,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:23,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:23,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:23,438 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:55:23,438 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:55:23,439 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@48bf455d[Count = 0] remaining members to acquire global barrier 2023-04-25 02:55:23,439 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:23,440 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:23,440 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:23,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:23,440 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:55:23,440 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:55:23,440 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:55:23,440 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:23,440 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:55:23,441 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:23,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 326 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54924 deadline: 1682391383436, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:23,445 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:55:23,445 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:23,445 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:23,445 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:55:23,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:23,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:23,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:23,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:23,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:23,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:23,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:23,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:23,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:23,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:23,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:55:23,447 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:55:23,447 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:55:23,447 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:55:23,447 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:55:23,448 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:55:23,448 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:55:23,449 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:23,449 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:23,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:23,449 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:23,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:23,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:23,449 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:23,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:23,449 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:23,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:23,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:23,450 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:23,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:23,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:23,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:23,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:23,451 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:23,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:23,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:23,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:23,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:23,453 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:23,453 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:23,453 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:23,453 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:23,453 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:23,453 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:23,453 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:23,453 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:23,453 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:55:23,453 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:23,453 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:55:23,453 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:55:23,453 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:23,453 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:23,454 DEBUG [Thread-2260] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:55:23,454 DEBUG [Thread-2260] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:55:23,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:23,454 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:23,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:23,461 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:23,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 334 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54884 deadline: 1682391383455, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:23,521 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=671 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/.tmp/B/87fec196952a43dba5e42ed48f704bb1 2023-04-25 02:55:23,554 DEBUG [Thread-2260] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:55:23,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:55:23,556 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:55:23,557 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:55:23,557 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:55:23,557 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:55:23,557 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:23,557 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:55:23,557 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:55:23,557 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:23,557 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:23,559 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:23,559 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:23,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:23,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:23,559 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:23,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:55:23,560 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:23,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:23,560 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:55:23,560 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:55:23,560 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:55:23,560 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:23,560 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:55:23,560 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:55:23,560 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:23,560 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:55:23,561 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:55:23,561 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:55:23,561 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. 2023-04-25 02:55:23,561 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. started... 2023-04-25 02:55:23,561 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. as already flushing 2023-04-25 02:55:23,561 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. 2023-04-25 02:55:23,561 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:55:23,561 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:55:23,561 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:23,561 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:55:23,561 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:55:23,563 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:23,563 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:23,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:23,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:23,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:23,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:23,564 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:23,564 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:55:23,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:23,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:23,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:23,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:23,565 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:55:23,565 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1afec5ec[Count = 0] remaining members to acquire global barrier 2023-04-25 02:55:23,565 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:55:23,566 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:23,567 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:23,567 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:23,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:23,567 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:55:23,567 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:55:23,567 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:55:23,568 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:55:23,568 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:23,568 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:55:23,568 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:23,569 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:55:23,569 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:55:23,569 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:55:23,569 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:55:23,569 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:55:23,569 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:55:23,569 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:55:23,570 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:23,570 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:23,570 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:23,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:23,570 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:23,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:23,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:23,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:23,571 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:23,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:23,571 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:23,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:23,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:23,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:23,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:23,572 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:23,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:23,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:23,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:23,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:23,576 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:23,576 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:55:23,576 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:55:23,576 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:55:23,577 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:23,577 DEBUG [Thread-2260] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:55:23,577 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:23,577 DEBUG [Thread-2260] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:55:23,577 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:23,577 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:23,577 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:23,577 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:23,577 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:23,584 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:23,584 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:23,584 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:23,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:23,585 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:23,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:23,677 DEBUG [Thread-2260] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:55:23,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:55:23,679 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:55:23,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:55:23,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:55:23,680 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:55:23,680 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:23,680 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:55:23,680 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:55:23,681 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:23,681 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:23,682 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:23,682 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:23,682 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:23,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:23,682 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:23,682 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:55:23,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:23,683 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:23,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:55:23,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:23,683 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:55:23,683 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:55:23,683 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:55:23,683 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:55:23,683 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:23,684 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:55:23,684 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:55:23,684 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. 2023-04-25 02:55:23,684 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. started... 2023-04-25 02:55:23,684 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:55:23,684 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. as already flushing 2023-04-25 02:55:23,684 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. 2023-04-25 02:55:23,684 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:55:23,684 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:55:23,684 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:23,684 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:55:23,684 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:55:23,685 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:23,686 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:23,686 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:55:23,688 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:23,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:23,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:23,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:23,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:23,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:23,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:23,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:23,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:23,691 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:55:23,691 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@76a02767[Count = 0] remaining members to acquire global barrier 2023-04-25 02:55:23,691 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:55:23,691 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:23,692 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:23,692 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:23,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:23,692 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:55:23,692 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:55:23,692 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:55:23,692 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:23,692 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:55:23,694 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:55:23,694 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:23,694 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:23,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:23,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:23,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:23,694 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:55:23,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:23,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:23,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:23,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:23,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:23,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:23,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:23,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:55:23,696 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:55:23,696 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:55:23,696 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:55:23,696 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:55:23,696 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:55:23,697 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:55:23,698 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:23,698 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:23,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:23,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:23,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:23,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:23,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:23,698 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:23,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:23,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:23,698 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:23,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:23,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:23,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:23,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:23,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:23,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:23,701 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:23,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:23,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:23,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:23,703 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:23,703 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:23,703 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:23,703 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:23,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:23,703 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:23,704 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:23,703 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:23,704 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:55:23,704 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:55:23,704 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:23,704 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:23,704 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:23,704 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:23,704 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:23,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:23,705 DEBUG [Thread-2260] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:55:23,705 DEBUG [Thread-2260] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:55:23,805 DEBUG [Thread-2260] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:55:23,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:55:23,808 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:55:23,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:55:23,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:55:23,809 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:55:23,809 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:23,809 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:55:23,809 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:55:23,809 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:23,809 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:23,810 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:23,810 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:23,810 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:23,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:23,811 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:23,811 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:55:23,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:23,811 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:23,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:55:23,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:23,812 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:55:23,812 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:55:23,812 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:55:23,812 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:55:23,812 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:23,812 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:55:23,812 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:55:23,812 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:55:23,812 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. 2023-04-25 02:55:23,812 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. started... 2023-04-25 02:55:23,812 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. as already flushing 2023-04-25 02:55:23,812 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. 2023-04-25 02:55:23,812 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:55:23,812 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:55:23,813 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:23,813 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:55:23,813 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:55:23,819 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:23,819 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:23,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:23,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:23,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:23,819 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:23,819 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:55:23,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:23,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:23,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:23,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:23,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:23,820 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:55:23,820 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2aad7a2b[Count = 0] remaining members to acquire global barrier 2023-04-25 02:55:23,820 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:55:23,820 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:23,821 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:23,821 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:23,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:23,822 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:55:23,822 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:23,822 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:55:23,822 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:55:23,822 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:55:23,823 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:23,823 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:55:23,823 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:23,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:23,823 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:55:23,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:23,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:23,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:23,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:23,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:23,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:23,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:23,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:23,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:23,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:55:23,825 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:55:23,825 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:55:23,825 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:55:23,825 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:55:23,825 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:55:23,826 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:55:23,830 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:23,830 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:23,830 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:23,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:23,830 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:23,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:23,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:23,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:23,830 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:23,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:23,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:23,830 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:23,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:23,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:23,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:23,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:23,832 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:23,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:23,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:23,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:23,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:23,833 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:23,833 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:23,833 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:23,833 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:23,833 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:23,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:23,833 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:23,833 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:55:23,834 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:55:23,833 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:23,834 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:55:23,834 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:23,834 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:23,834 DEBUG [Thread-2260] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:55:23,834 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:23,834 DEBUG [Thread-2260] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:55:23,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:23,834 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:23,834 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:23,932 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=671 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/.tmp/C/a4435fdcf26140528897c769aba75e92 2023-04-25 02:55:23,934 DEBUG [Thread-2260] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:55:23,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:55:23,936 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/.tmp/A/d31fd5cb265748fda29a79ca2593bc32 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/A/d31fd5cb265748fda29a79ca2593bc32 2023-04-25 02:55:23,936 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:55:23,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:55:23,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:55:23,938 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:55:23,938 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:23,938 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:55:23,938 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:55:23,938 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:23,938 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:23,939 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:23,939 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:23,939 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:23,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:23,940 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:23,940 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/A/d31fd5cb265748fda29a79ca2593bc32, entries=200, sequenceid=671, filesize=32.3 K 2023-04-25 02:55:23,940 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:55:23,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:23,940 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:23,941 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/.tmp/B/87fec196952a43dba5e42ed48f704bb1 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/B/87fec196952a43dba5e42ed48f704bb1 2023-04-25 02:55:23,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:55:23,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:23,941 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:55:23,941 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:55:23,941 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:55:23,941 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:55:23,941 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:23,941 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:55:23,941 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:55:23,942 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:55:23,942 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. 2023-04-25 02:55:23,942 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. started... 2023-04-25 02:55:23,942 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. as already flushing 2023-04-25 02:55:23,942 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. 2023-04-25 02:55:23,942 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:55:23,942 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:55:23,942 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:23,942 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:55:23,942 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:55:23,944 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:23,944 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:23,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:23,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:23,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:23,944 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:23,944 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:55:23,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:23,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:23,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:23,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:23,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:23,946 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:55:23,946 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@188a478e[Count = 0] remaining members to acquire global barrier 2023-04-25 02:55:23,946 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:55:23,946 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:23,946 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/B/87fec196952a43dba5e42ed48f704bb1, entries=150, sequenceid=671, filesize=11.9 K 2023-04-25 02:55:23,947 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/.tmp/C/a4435fdcf26140528897c769aba75e92 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/C/a4435fdcf26140528897c769aba75e92 2023-04-25 02:55:23,948 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:23,948 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:23,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:23,948 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:55:23,948 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:55:23,948 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:55:23,948 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:23,948 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:55:23,949 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:55:23,949 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:23,949 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:23,950 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:55:23,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:23,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:23,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:23,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:23,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:23,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:23,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:23,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:23,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:23,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:23,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:55:23,955 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:55:23,955 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:55:23,955 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:55:23,955 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:55:23,955 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:55:23,955 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:55:23,957 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:23,957 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:23,957 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:23,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:23,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:23,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:23,957 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:23,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:23,957 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:23,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:23,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:23,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:23,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:23,958 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:23,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:23,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:23,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:23,959 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:23,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:23,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:23,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:23,961 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:23,961 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:23,961 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:23,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:23,962 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:23,962 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:23,962 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:23,962 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:23,962 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:23,962 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:23,962 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:23,962 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:23,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:23,962 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:55:23,962 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:55:23,962 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:23,962 DEBUG [Thread-2260] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:55:23,962 DEBUG [Thread-2260] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:55:23,968 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/C/a4435fdcf26140528897c769aba75e92, entries=150, sequenceid=671, filesize=11.9 K 2023-04-25 02:55:23,969 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~404.34 KB/414048, currentSize=40.25 KB/41220 for 9d5b411b5fe67e5bb007f4d36e3e11e5 in 1700ms, sequenceid=671, compaction requested=true 2023-04-25 02:55:23,969 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 9d5b411b5fe67e5bb007f4d36e3e11e5: 2023-04-25 02:55:23,969 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 02:55:23,969 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 02:55:23,969 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 8 store files, 0 compacting, 8 eligible, 16 blocking 2023-04-25 02:55:23,969 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 8 store files, 0 compacting, 8 eligible, 16 blocking 2023-04-25 02:55:23,969 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 02:55:23,973 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 8 files of size 95304 starting at candidate #0 after considering 21 permutations with 21 in ratio 2023-04-25 02:55:23,973 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HStore(1912): 9d5b411b5fe67e5bb007f4d36e3e11e5/B is initiating minor compaction (all files) 2023-04-25 02:55:23,973 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 9d5b411b5fe67e5bb007f4d36e3e11e5/B in TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. 2023-04-25 02:55:23,973 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/B/97647b20a62d4a64aa99e34eb1fda79d, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/B/2626d6f8ce5447be9decb521b7efb663, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/B/d317e1976c45442496ee612abf0ac5c3, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/B/7e0bb30ba5ae4d09b3b44e47268e1278, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/B/b00219107a614a578ce1ff9ef9b51c3f, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/B/66a4d84d12e24db0abb6101e481f225e, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/B/102dde04601f4b6491d366c3895677e0, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/B/87fec196952a43dba5e42ed48f704bb1] into tmpdir=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/.tmp, totalSize=93.1 K 2023-04-25 02:55:23,973 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 8 files of size 229748 starting at candidate #0 after considering 21 permutations with 21 in ratio 2023-04-25 02:55:23,973 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HStore(1912): 9d5b411b5fe67e5bb007f4d36e3e11e5/A is initiating minor compaction (all files) 2023-04-25 02:55:23,973 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 9d5b411b5fe67e5bb007f4d36e3e11e5/A in TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. 2023-04-25 02:55:23,974 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/A/98e9d45bf15a4411ba742e90717013e8, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/A/d819bdc04029412bbefde1fb19bd9c7a, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/A/6c6b168e277349f9a7c5d63214ad32a1, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/A/090c421124fe44b98ff74ae2318d0481, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/A/140f06ca977e4666814670510e7c6fa2, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/A/a3a34b8e3d4941d1977b128b8d773072, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/A/f3c5113141344f78a252d9fe29299a30, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/A/d31fd5cb265748fda29a79ca2593bc32] into tmpdir=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/.tmp, totalSize=224.4 K 2023-04-25 02:55:23,974 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting 97647b20a62d4a64aa99e34eb1fda79d, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=216, earliestPutTs=1682391314548 2023-04-25 02:55:23,974 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting 98e9d45bf15a4411ba742e90717013e8, keycount=150, bloomtype=ROW, size=25.7 K, encoding=NONE, compression=NONE, seqNum=216, earliestPutTs=1682391314548 2023-04-25 02:55:23,974 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting 2626d6f8ce5447be9decb521b7efb663, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=439, earliestPutTs=1682391316047 2023-04-25 02:55:23,975 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting d819bdc04029412bbefde1fb19bd9c7a, keycount=250, bloomtype=ROW, size=39.2 K, encoding=NONE, compression=NONE, seqNum=439, earliestPutTs=1682391316033 2023-04-25 02:55:23,975 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting 6c6b168e277349f9a7c5d63214ad32a1, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=454, earliestPutTs=1682391316086 2023-04-25 02:55:23,975 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting d317e1976c45442496ee612abf0ac5c3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=454, earliestPutTs=1682391316086 2023-04-25 02:55:23,975 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting 090c421124fe44b98ff74ae2318d0481, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=474, earliestPutTs=1682391317275 2023-04-25 02:55:23,975 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting 7e0bb30ba5ae4d09b3b44e47268e1278, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=474, earliestPutTs=1682391317275 2023-04-25 02:55:23,976 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting 140f06ca977e4666814670510e7c6fa2, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=486, earliestPutTs=1682391318425 2023-04-25 02:55:23,976 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting b00219107a614a578ce1ff9ef9b51c3f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=486, earliestPutTs=1682391318425 2023-04-25 02:55:23,976 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting a3a34b8e3d4941d1977b128b8d773072, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=506, earliestPutTs=1682391319098 2023-04-25 02:55:23,976 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting 66a4d84d12e24db0abb6101e481f225e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=506, earliestPutTs=1682391319098 2023-04-25 02:55:23,977 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting f3c5113141344f78a252d9fe29299a30, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=513, earliestPutTs=1682391321290 2023-04-25 02:55:23,977 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting 102dde04601f4b6491d366c3895677e0, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=513, earliestPutTs=1682391321359 2023-04-25 02:55:23,977 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting d31fd5cb265748fda29a79ca2593bc32, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=671, earliestPutTs=1682391322241 2023-04-25 02:55:23,977 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting 87fec196952a43dba5e42ed48f704bb1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=671, earliestPutTs=1682391322241 2023-04-25 02:55:23,992 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] throttle.PressureAwareThroughputController(145): 9d5b411b5fe67e5bb007f4d36e3e11e5#B#compaction#354 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-25 02:55:24,030 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] throttle.PressureAwareThroughputController(145): 9d5b411b5fe67e5bb007f4d36e3e11e5#A#compaction#355 average throughput is 0.56 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-25 02:55:24,033 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/.tmp/B/65409474b81645d5985504057ff9b988 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/B/65409474b81645d5985504057ff9b988 2023-04-25 02:55:24,038 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HStore(1652): Completed compaction of 8 (all) file(s) in 9d5b411b5fe67e5bb007f4d36e3e11e5/B of 9d5b411b5fe67e5bb007f4d36e3e11e5 into 65409474b81645d5985504057ff9b988(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 02:55:24,038 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9d5b411b5fe67e5bb007f4d36e3e11e5: 2023-04-25 02:55:24,038 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5., storeName=9d5b411b5fe67e5bb007f4d36e3e11e5/B, priority=8, startTime=1682391323969; duration=0sec 2023-04-25 02:55:24,038 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 02:55:24,038 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 8 store files, 0 compacting, 8 eligible, 16 blocking 2023-04-25 02:55:24,041 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 8 files of size 95304 starting at candidate #0 after considering 21 permutations with 21 in ratio 2023-04-25 02:55:24,042 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HStore(1912): 9d5b411b5fe67e5bb007f4d36e3e11e5/C is initiating minor compaction (all files) 2023-04-25 02:55:24,042 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 9d5b411b5fe67e5bb007f4d36e3e11e5/C in TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. 2023-04-25 02:55:24,042 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/C/e9523fdf37964c0ab71c8aff66e14f9b, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/C/0a7842e5f47a403f9545049720c83770, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/C/4ba2891cb65b44c99fd2da78a0977bf8, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/C/67b5bda6cb9b4513ac61fd00f7b73765, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/C/ed72f11460d34a31b8dae931145a746a, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/C/123c0380fc2e44919ac8b3efc9a443bf, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/C/616aca292ee64a4cb7f7bcce4a25a9e3, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/C/a4435fdcf26140528897c769aba75e92] into tmpdir=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/.tmp, totalSize=93.1 K 2023-04-25 02:55:24,042 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting e9523fdf37964c0ab71c8aff66e14f9b, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=216, earliestPutTs=1682391314548 2023-04-25 02:55:24,043 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting 0a7842e5f47a403f9545049720c83770, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=439, earliestPutTs=1682391316047 2023-04-25 02:55:24,043 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting 4ba2891cb65b44c99fd2da78a0977bf8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=454, earliestPutTs=1682391316086 2023-04-25 02:55:24,044 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting 67b5bda6cb9b4513ac61fd00f7b73765, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=474, earliestPutTs=1682391317275 2023-04-25 02:55:24,044 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting ed72f11460d34a31b8dae931145a746a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=486, earliestPutTs=1682391318425 2023-04-25 02:55:24,044 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting 123c0380fc2e44919ac8b3efc9a443bf, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=506, earliestPutTs=1682391319098 2023-04-25 02:55:24,045 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting 616aca292ee64a4cb7f7bcce4a25a9e3, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=513, earliestPutTs=1682391321359 2023-04-25 02:55:24,045 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting a4435fdcf26140528897c769aba75e92, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=671, earliestPutTs=1682391322241 2023-04-25 02:55:24,060 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] throttle.PressureAwareThroughputController(145): 9d5b411b5fe67e5bb007f4d36e3e11e5#C#compaction#356 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-25 02:55:24,063 DEBUG [Thread-2260] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:55:24,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:55:24,065 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:55:24,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:55:24,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:55:24,068 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:55:24,068 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:24,068 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:55:24,068 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:55:24,069 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:24,069 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:24,071 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:24,071 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:24,071 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:24,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:24,072 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:24,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:24,072 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:55:24,072 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:24,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:55:24,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:24,072 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:55:24,088 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:55:24,088 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:55:24,088 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:55:24,088 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:24,088 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:55:24,089 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:55:24,089 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:55:24,089 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. 2023-04-25 02:55:24,089 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. started... 2023-04-25 02:55:24,090 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 9d5b411b5fe67e5bb007f4d36e3e11e5 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-04-25 02:55:24,092 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9d5b411b5fe67e5bb007f4d36e3e11e5, store=A 2023-04-25 02:55:24,092 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:55:24,092 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9d5b411b5fe67e5bb007f4d36e3e11e5, store=B 2023-04-25 02:55:24,092 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:55:24,092 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9d5b411b5fe67e5bb007f4d36e3e11e5, store=C 2023-04-25 02:55:24,092 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:55:24,094 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/.tmp/C/a21351296fe64c72b088bc29363c0672 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/C/a21351296fe64c72b088bc29363c0672 2023-04-25 02:55:24,101 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HStore(1652): Completed compaction of 8 (all) file(s) in 9d5b411b5fe67e5bb007f4d36e3e11e5/C of 9d5b411b5fe67e5bb007f4d36e3e11e5 into a21351296fe64c72b088bc29363c0672(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 02:55:24,101 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9d5b411b5fe67e5bb007f4d36e3e11e5: 2023-04-25 02:55:24,101 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5., storeName=9d5b411b5fe67e5bb007f4d36e3e11e5/C, priority=8, startTime=1682391323969; duration=0sec 2023-04-25 02:55:24,101 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 02:55:24,147 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304254a851860ab344ca49f9d35382604779a to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304254a851860ab344ca49f9d35382604779a 2023-04-25 02:55:24,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(9158): Flush requested on 9d5b411b5fe67e5bb007f4d36e3e11e5 2023-04-25 02:55:24,481 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. as already flushing 2023-04-25 02:55:24,570 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=681, memsize=13.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/.tmp/A/96da6605b9c9489299d34647ebed19b1 2023-04-25 02:55:24,615 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682391324584, max timestamp=1682391324614Num uniques 859; 2023-04-25 02:55:24,615 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-25 02:55:24,616 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682391324584, max timestamp=1682391324614Num uniques 859; 2023-04-25 02:55:24,616 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-25 02:55:24,616 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682391324584, max timestamp=1682391324614Num uniques 859; 2023-04-25 02:55:24,616 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-25 02:55:24,722 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1682391324668, max timestamp=1682391324718Num uniques 1018; 2023-04-25 02:55:24,722 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-25 02:55:24,724 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1682391324668, max timestamp=1682391324718Num uniques 1018; 2023-04-25 02:55:24,724 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-25 02:55:24,725 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1682391324668, max timestamp=1682391324718Num uniques 1018; 2023-04-25 02:55:24,725 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-25 02:55:24,809 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1682391324782, max timestamp=1682391324806Num uniques 1077; 2023-04-25 02:55:24,809 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1682391324782, max timestamp=1682391324806Num uniques 1077; 2023-04-25 02:55:24,809 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=10392, suffix heap size=139976,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=227 2023-04-25 02:55:24,811 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1682391324782, max timestamp=1682391324806Num uniques 1077; 2023-04-25 02:55:24,812 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=10392, suffix heap size=139976,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=227 2023-04-25 02:55:24,809 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=10392, suffix heap size=139976,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=227 2023-04-25 02:55:24,865 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/.tmp/A/3e89e4af11aa40ecb9fa4f8e10d6c59e as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/A/3e89e4af11aa40ecb9fa4f8e10d6c59e 2023-04-25 02:55:24,879 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 8 (all) file(s) in 9d5b411b5fe67e5bb007f4d36e3e11e5/A of 9d5b411b5fe67e5bb007f4d36e3e11e5 into 3e89e4af11aa40ecb9fa4f8e10d6c59e(size=26.1 K), total size for store is 26.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 02:55:24,879 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9d5b411b5fe67e5bb007f4d36e3e11e5: 2023-04-25 02:55:24,879 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5., storeName=9d5b411b5fe67e5bb007f4d36e3e11e5/A, priority=8, startTime=1682391323969; duration=0sec 2023-04-25 02:55:24,879 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 02:55:24,911 DEBUG [Thread-2263] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1d7ba242 to 127.0.0.1:62054 2023-04-25 02:55:24,911 DEBUG [Thread-2263] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-25 02:55:24,912 DEBUG [Thread-2265] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3e079da1 to 127.0.0.1:62054 2023-04-25 02:55:24,912 DEBUG [Thread-2265] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-25 02:55:24,913 DEBUG [Thread-2261] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1ec12b02 to 127.0.0.1:62054 2023-04-25 02:55:24,914 DEBUG [Thread-2261] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-25 02:55:24,917 DEBUG [Thread-2256] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x10377b50 to 127.0.0.1:62054 2023-04-25 02:55:24,917 DEBUG [Thread-2252] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3620ee35 to 127.0.0.1:62054 2023-04-25 02:55:24,917 DEBUG [Thread-2256] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-25 02:55:24,917 DEBUG [Thread-2252] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-25 02:55:24,924 DEBUG [Thread-2269] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7bec116b to 127.0.0.1:62054 2023-04-25 02:55:24,924 DEBUG [Thread-2269] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-25 02:55:24,927 DEBUG [Thread-2267] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x159c5e0c to 127.0.0.1:62054 2023-04-25 02:55:24,927 DEBUG [Thread-2267] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-25 02:55:25,000 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=681 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/.tmp/B/f9bfad94978d472e96b6a5a4b4c5b9dd 2023-04-25 02:55:25,411 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=681 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/.tmp/C/7efb1cb3804947ad8337934202335b51 2023-04-25 02:55:25,416 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/.tmp/A/96da6605b9c9489299d34647ebed19b1 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/A/96da6605b9c9489299d34647ebed19b1 2023-04-25 02:55:25,419 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/A/96da6605b9c9489299d34647ebed19b1, entries=100, sequenceid=681, filesize=18.6 K 2023-04-25 02:55:25,420 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/.tmp/B/f9bfad94978d472e96b6a5a4b4c5b9dd as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/B/f9bfad94978d472e96b6a5a4b4c5b9dd 2023-04-25 02:55:25,422 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/B/f9bfad94978d472e96b6a5a4b4c5b9dd, entries=100, sequenceid=681, filesize=9.5 K 2023-04-25 02:55:25,423 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/.tmp/C/7efb1cb3804947ad8337934202335b51 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/C/7efb1cb3804947ad8337934202335b51 2023-04-25 02:55:25,425 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/C/7efb1cb3804947ad8337934202335b51, entries=100, sequenceid=681, filesize=9.5 K 2023-04-25 02:55:25,426 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=134.18 KB/137400 for 9d5b411b5fe67e5bb007f4d36e3e11e5 in 1336ms, sequenceid=681, compaction requested=false 2023-04-25 02:55:25,426 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9d5b411b5fe67e5bb007f4d36e3e11e5: 2023-04-25 02:55:25,426 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. 2023-04-25 02:55:25,426 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:55:25,426 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:55:25,426 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:25,426 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:55:25,426 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:55:25,428 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:25,428 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:25,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:25,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:25,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:25,428 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:25,428 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:55:25,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:25,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:25,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:25,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:25,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:25,429 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:55:25,429 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@743720d6[Count = 0] remaining members to acquire global barrier 2023-04-25 02:55:25,429 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:55:25,429 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:25,430 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:25,430 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:25,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:25,430 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:55:25,430 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:25,430 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:55:25,430 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:55:25,430 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:55:25,432 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:55:25,432 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:25,432 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:25,432 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:55:25,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:25,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:25,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:25,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:25,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:25,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:25,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:25,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:25,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:25,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:25,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:55:25,434 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:55:25,434 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:55:25,434 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:55:25,434 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:55:25,434 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:55:25,434 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:55:25,436 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:25,436 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:25,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:25,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:25,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:25,436 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:25,436 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:25,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:25,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:25,436 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:25,436 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:25,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:25,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:25,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:25,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:25,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:25,437 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:25,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:25,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:25,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:25,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:25,439 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:25,439 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:25,439 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:25,439 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:25,439 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:25,439 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:55:25,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:25,439 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:55:25,439 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:55:25,439 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:25,439 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:25,439 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:25,439 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:25,439 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:25,439 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:25,439 DEBUG [Thread-2260] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:55:25,439 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:25,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:25,440 DEBUG [Thread-2260] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:55:25,540 DEBUG [Thread-2260] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:55:25,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:55:30,251 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(9158): Flush requested on 9d5b411b5fe67e5bb007f4d36e3e11e5 2023-04-25 02:55:30,251 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 9d5b411b5fe67e5bb007f4d36e3e11e5 3/3 column families, dataSize=140.89 KB heapSize=398.53 KB 2023-04-25 02:55:30,251 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9d5b411b5fe67e5bb007f4d36e3e11e5, store=A 2023-04-25 02:55:30,251 DEBUG [Thread-2258] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4366fcdd to 127.0.0.1:62054 2023-04-25 02:55:30,251 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 02:55:30,251 DEBUG [Thread-2258] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-25 02:55:30,251 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9d5b411b5fe67e5bb007f4d36e3e11e5, store=B 2023-04-25 02:55:30,251 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 02:55:30,251 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9d5b411b5fe67e5bb007f4d36e3e11e5, store=C 2023-04-25 02:55:30,251 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 02:55:30,266 DEBUG [Thread-2254] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4fe1db6a to 127.0.0.1:62054 2023-04-25 02:55:30,266 DEBUG [Thread-2254] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-25 02:55:30,302 DEBUG [Thread-2250] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x71b5058b to 127.0.0.1:62054 2023-04-25 02:55:30,302 DEBUG [Thread-2250] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-25 02:55:30,302 INFO [Listener at localhost/41915] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-25 02:55:30,302 INFO [Listener at localhost/41915] hbase.AcidGuaranteesTestTool(394): wrote 40 2023-04-25 02:55:30,302 INFO [Listener at localhost/41915] hbase.AcidGuaranteesTestTool(394): wrote 288 2023-04-25 02:55:30,302 INFO [Listener at localhost/41915] hbase.AcidGuaranteesTestTool(394): wrote 39 2023-04-25 02:55:30,302 INFO [Listener at localhost/41915] hbase.AcidGuaranteesTestTool(394): wrote 279 2023-04-25 02:55:30,303 INFO [Listener at localhost/41915] hbase.AcidGuaranteesTestTool(394): wrote 37 2023-04-25 02:55:30,303 INFO [Listener at localhost/41915] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-25 02:55:30,303 INFO [Listener at localhost/41915] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-25 02:55:30,303 INFO [Listener at localhost/41915] hbase.AcidGuaranteesTestTool(402): scanned 1725 2023-04-25 02:55:30,303 INFO [Listener at localhost/41915] hbase.AcidGuaranteesTestTool(403): verified 5175 rows 2023-04-25 02:55:30,303 INFO [Listener at localhost/41915] hbase.AcidGuaranteesTestTool(402): scanned 1727 2023-04-25 02:55:30,303 INFO [Listener at localhost/41915] hbase.AcidGuaranteesTestTool(403): verified 5181 rows 2023-04-25 02:55:30,303 INFO [Listener at localhost/41915] hbase.AcidGuaranteesTestTool(402): scanned 1736 2023-04-25 02:55:30,303 INFO [Listener at localhost/41915] hbase.AcidGuaranteesTestTool(403): verified 5207 rows 2023-04-25 02:55:30,303 INFO [Listener at localhost/41915] hbase.AcidGuaranteesTestTool(402): scanned 1737 2023-04-25 02:55:30,303 INFO [Listener at localhost/41915] hbase.AcidGuaranteesTestTool(403): verified 5207 rows 2023-04-25 02:55:30,303 INFO [Listener at localhost/41915] hbase.AcidGuaranteesTestTool(402): scanned 1738 2023-04-25 02:55:30,303 INFO [Listener at localhost/41915] hbase.AcidGuaranteesTestTool(403): verified 5214 rows 2023-04-25 02:55:30,303 INFO [Listener at localhost/41915] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-25 02:55:30,303 DEBUG [Listener at localhost/41915] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x38341134 to 127.0.0.1:62054 2023-04-25 02:55:30,303 DEBUG [Listener at localhost/41915] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-25 02:55:30,306 INFO [Listener at localhost/41915] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-25 02:55:30,307 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-04-25 02:55:30,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure2.ProcedureExecutor(1029): Stored pid=43, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-25 02:55:30,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-25 02:55:30,311 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682391330311"}]},"ts":"1682391330311"} 2023-04-25 02:55:30,312 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-25 02:55:30,313 INFO [PEWorker-1] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-25 02:55:30,314 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=44, ppid=43, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=9d5b411b5fe67e5bb007f4d36e3e11e5, UNASSIGN}] 2023-04-25 02:55:30,315 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, ppid=43, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=9d5b411b5fe67e5bb007f4d36e3e11e5, UNASSIGN 2023-04-25 02:55:30,315 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=44 updating hbase:meta row=9d5b411b5fe67e5bb007f4d36e3e11e5, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:30,316 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682391330315"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682391330315"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682391330315"}]},"ts":"1682391330315"} 2023-04-25 02:55:30,316 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=45, ppid=44, state=RUNNABLE; CloseRegionProcedure 9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023}] 2023-04-25 02:55:30,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-25 02:55:30,468 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 9d5b411b5fe67e5bb007f4d36e3e11e5 2023-04-25 02:55:30,468 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 9d5b411b5fe67e5bb007f4d36e3e11e5, disabling compactions & flushes 2023-04-25 02:55:30,468 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1858): waiting for 0 compactions & cache flush to complete for region TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. 2023-04-25 02:55:30,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-25 02:55:30,665 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230425bbc6ab684edc4a4eb8032431c1832142 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425bbc6ab684edc4a4eb8032431c1832142 2023-04-25 02:55:30,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-25 02:55:31,070 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=754, memsize=47.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/.tmp/A/3640d17ea7ee420bb4b79e310beeb6c8 2023-04-25 02:55:31,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-25 02:55:31,479 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=754 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/.tmp/B/b9da6db969584ed5912fd6cc68b87803 2023-04-25 02:55:31,888 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=754 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/.tmp/C/9bafd9e804ff451188b2045fc8468d49 2023-04-25 02:55:31,891 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/.tmp/A/3640d17ea7ee420bb4b79e310beeb6c8 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/A/3640d17ea7ee420bb4b79e310beeb6c8 2023-04-25 02:55:31,894 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/A/3640d17ea7ee420bb4b79e310beeb6c8, entries=150, sequenceid=754, filesize=25.4 K 2023-04-25 02:55:31,894 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/.tmp/B/b9da6db969584ed5912fd6cc68b87803 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/B/b9da6db969584ed5912fd6cc68b87803 2023-04-25 02:55:31,897 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/B/b9da6db969584ed5912fd6cc68b87803, entries=150, sequenceid=754, filesize=11.9 K 2023-04-25 02:55:31,898 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/.tmp/C/9bafd9e804ff451188b2045fc8468d49 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/C/9bafd9e804ff451188b2045fc8468d49 2023-04-25 02:55:31,900 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/C/9bafd9e804ff451188b2045fc8468d49, entries=150, sequenceid=754, filesize=11.9 K 2023-04-25 02:55:31,901 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~398.48 KB/408048, currentSize=13.42 KB/13740 for 9d5b411b5fe67e5bb007f4d36e3e11e5 in 1650ms, sequenceid=754, compaction requested=true 2023-04-25 02:55:31,901 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 9d5b411b5fe67e5bb007f4d36e3e11e5: 2023-04-25 02:55:31,901 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. 2023-04-25 02:55:31,901 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 02:55:31,901 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. 2023-04-25 02:55:31,901 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. because compaction request was cancelled 2023-04-25 02:55:31,901 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. because compaction request was cancelled 2023-04-25 02:55:31,901 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. after waiting 0 ms 2023-04-25 02:55:31,901 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 02:55:31,901 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. 2023-04-25 02:55:31,901 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 02:55:31,901 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 9d5b411b5fe67e5bb007f4d36e3e11e5 3/3 column families, dataSize=13.42 KB heapSize=40.59 KB 2023-04-25 02:55:31,901 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. because compaction request was cancelled 2023-04-25 02:55:31,902 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9d5b411b5fe67e5bb007f4d36e3e11e5, store=A 2023-04-25 02:55:31,902 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:55:31,902 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9d5b411b5fe67e5bb007f4d36e3e11e5, store=B 2023-04-25 02:55:31,902 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:55:31,902 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9d5b411b5fe67e5bb007f4d36e3e11e5, store=C 2023-04-25 02:55:31,902 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:55:32,311 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230425e4497fccd6e84b449d86d1b144cd9bbe to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425e4497fccd6e84b449d86d1b144cd9bbe 2023-04-25 02:55:32,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-25 02:55:32,716 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=759, memsize=4.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/.tmp/A/9209e8f26b504f3186755a8e6d2f7847 2023-04-25 02:55:33,124 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=4.47 KB at sequenceid=759 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/.tmp/B/5ecad234252e4aa4ae9bb80cbd0f985b 2023-04-25 02:55:33,532 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=4.47 KB at sequenceid=759 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/.tmp/C/dbccc78f5b4f406eb56b73e4ce34c9c2 2023-04-25 02:55:33,536 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/.tmp/A/9209e8f26b504f3186755a8e6d2f7847 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/A/9209e8f26b504f3186755a8e6d2f7847 2023-04-25 02:55:33,540 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/A/9209e8f26b504f3186755a8e6d2f7847, entries=100, sequenceid=759, filesize=18.6 K 2023-04-25 02:55:33,541 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/.tmp/B/5ecad234252e4aa4ae9bb80cbd0f985b as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/B/5ecad234252e4aa4ae9bb80cbd0f985b 2023-04-25 02:55:33,544 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/B/5ecad234252e4aa4ae9bb80cbd0f985b, entries=100, sequenceid=759, filesize=9.5 K 2023-04-25 02:55:33,545 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/.tmp/C/dbccc78f5b4f406eb56b73e4ce34c9c2 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/C/dbccc78f5b4f406eb56b73e4ce34c9c2 2023-04-25 02:55:33,548 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/C/dbccc78f5b4f406eb56b73e4ce34c9c2, entries=100, sequenceid=759, filesize=9.5 K 2023-04-25 02:55:33,549 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~13.42 KB/13740, heapSize ~40.55 KB/41520, currentSize=0 B/0 for 9d5b411b5fe67e5bb007f4d36e3e11e5 in 1648ms, sequenceid=759, compaction requested=true 2023-04-25 02:55:33,570 DEBUG [StoreCloser-TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/A/ddd35b6b0e81423c9de1386458364536, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/A/7bf464a021a04faebc51627281a358d2, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/A/93a2909c726741f0b881413a2c5cf7a5, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/A/255914514a5e4a768a16f509fcf81a08, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/A/fde37d724a4447e189b46f2fe3ded178, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/A/0819a75a91174ab38d49623fbe728436, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/A/b82f6c1510c545ba8f5be887bb396bbe, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/A/aa223b4266764748a286c96dc8f5f842, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/A/82ca2bf000e646d9b901407e5783065f, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/A/a15c8b364906459f9302cbcc776860f8, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/A/78b7d87538f14c738296e7cadb416c9e, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/A/8cc484f08b9d4fbb92fa75c9c9de144c, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/A/98e9d45bf15a4411ba742e90717013e8, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/A/d819bdc04029412bbefde1fb19bd9c7a, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/A/6c6b168e277349f9a7c5d63214ad32a1, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/A/090c421124fe44b98ff74ae2318d0481, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/A/140f06ca977e4666814670510e7c6fa2, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/A/a3a34b8e3d4941d1977b128b8d773072, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/A/f3c5113141344f78a252d9fe29299a30, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/A/d31fd5cb265748fda29a79ca2593bc32] to archive 2023-04-25 02:55:33,571 DEBUG [StoreCloser-TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-25 02:55:33,573 DEBUG [StoreCloser-TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/A/ddd35b6b0e81423c9de1386458364536 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/A/ddd35b6b0e81423c9de1386458364536 2023-04-25 02:55:33,574 DEBUG [StoreCloser-TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/A/7bf464a021a04faebc51627281a358d2 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/A/7bf464a021a04faebc51627281a358d2 2023-04-25 02:55:33,574 DEBUG [StoreCloser-TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/A/93a2909c726741f0b881413a2c5cf7a5 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/A/93a2909c726741f0b881413a2c5cf7a5 2023-04-25 02:55:33,575 DEBUG [StoreCloser-TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/A/255914514a5e4a768a16f509fcf81a08 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/A/255914514a5e4a768a16f509fcf81a08 2023-04-25 02:55:33,576 DEBUG [StoreCloser-TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/A/fde37d724a4447e189b46f2fe3ded178 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/A/fde37d724a4447e189b46f2fe3ded178 2023-04-25 02:55:33,577 DEBUG [StoreCloser-TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/A/0819a75a91174ab38d49623fbe728436 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/A/0819a75a91174ab38d49623fbe728436 2023-04-25 02:55:33,578 DEBUG [StoreCloser-TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/A/b82f6c1510c545ba8f5be887bb396bbe to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/A/b82f6c1510c545ba8f5be887bb396bbe 2023-04-25 02:55:33,578 DEBUG [StoreCloser-TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/A/aa223b4266764748a286c96dc8f5f842 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/A/aa223b4266764748a286c96dc8f5f842 2023-04-25 02:55:33,579 DEBUG [StoreCloser-TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/A/82ca2bf000e646d9b901407e5783065f to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/A/82ca2bf000e646d9b901407e5783065f 2023-04-25 02:55:33,580 DEBUG [StoreCloser-TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/A/a15c8b364906459f9302cbcc776860f8 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/A/a15c8b364906459f9302cbcc776860f8 2023-04-25 02:55:33,581 DEBUG [StoreCloser-TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/A/78b7d87538f14c738296e7cadb416c9e to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/A/78b7d87538f14c738296e7cadb416c9e 2023-04-25 02:55:33,581 DEBUG [StoreCloser-TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/A/8cc484f08b9d4fbb92fa75c9c9de144c to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/A/8cc484f08b9d4fbb92fa75c9c9de144c 2023-04-25 02:55:33,582 DEBUG [StoreCloser-TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/A/98e9d45bf15a4411ba742e90717013e8 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/A/98e9d45bf15a4411ba742e90717013e8 2023-04-25 02:55:33,583 DEBUG [StoreCloser-TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/A/d819bdc04029412bbefde1fb19bd9c7a to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/A/d819bdc04029412bbefde1fb19bd9c7a 2023-04-25 02:55:33,584 DEBUG [StoreCloser-TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/A/6c6b168e277349f9a7c5d63214ad32a1 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/A/6c6b168e277349f9a7c5d63214ad32a1 2023-04-25 02:55:33,585 DEBUG [StoreCloser-TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/A/090c421124fe44b98ff74ae2318d0481 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/A/090c421124fe44b98ff74ae2318d0481 2023-04-25 02:55:33,585 DEBUG [StoreCloser-TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/A/140f06ca977e4666814670510e7c6fa2 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/A/140f06ca977e4666814670510e7c6fa2 2023-04-25 02:55:33,586 DEBUG [StoreCloser-TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/A/a3a34b8e3d4941d1977b128b8d773072 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/A/a3a34b8e3d4941d1977b128b8d773072 2023-04-25 02:55:33,587 DEBUG [StoreCloser-TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/A/f3c5113141344f78a252d9fe29299a30 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/A/f3c5113141344f78a252d9fe29299a30 2023-04-25 02:55:33,588 DEBUG [StoreCloser-TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/A/d31fd5cb265748fda29a79ca2593bc32 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/A/d31fd5cb265748fda29a79ca2593bc32 2023-04-25 02:55:33,590 DEBUG [StoreCloser-TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/B/1db215561a814ddda5f10046d1fa87c9, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/B/9c43d51c27cc4b3dabae028f220942cb, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/B/d9cc943a70804409856182741fa323fc, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/B/4452115a19bd4f5e83148150b3a27076, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/B/4f7a577a5df44d17bb388655eb7aff52, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/B/886195b72dee4da182fa19a6b333d181, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/B/765c196572fd4dc9999fca242f93b77b, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/B/3703b5da4c594ed68d58845cfd8cb0ad, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/B/3c2de4efd527437f9f4188dc83ab23f8, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/B/f2fd638cc44b47939bbbc6e7dbea7fcd, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/B/66c99592f17d4d0ca68c50099d8196ad, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/B/97647b20a62d4a64aa99e34eb1fda79d, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/B/c51b3153725541ca879c6a927da493eb, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/B/2626d6f8ce5447be9decb521b7efb663, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/B/d317e1976c45442496ee612abf0ac5c3, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/B/7e0bb30ba5ae4d09b3b44e47268e1278, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/B/b00219107a614a578ce1ff9ef9b51c3f, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/B/66a4d84d12e24db0abb6101e481f225e, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/B/102dde04601f4b6491d366c3895677e0, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/B/87fec196952a43dba5e42ed48f704bb1] to archive 2023-04-25 02:55:33,591 DEBUG [StoreCloser-TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-25 02:55:33,592 DEBUG [StoreCloser-TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/B/1db215561a814ddda5f10046d1fa87c9 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/B/1db215561a814ddda5f10046d1fa87c9 2023-04-25 02:55:33,593 DEBUG [StoreCloser-TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/B/9c43d51c27cc4b3dabae028f220942cb to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/B/9c43d51c27cc4b3dabae028f220942cb 2023-04-25 02:55:33,594 DEBUG [StoreCloser-TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/B/d9cc943a70804409856182741fa323fc to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/B/d9cc943a70804409856182741fa323fc 2023-04-25 02:55:33,595 DEBUG [StoreCloser-TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/B/4452115a19bd4f5e83148150b3a27076 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/B/4452115a19bd4f5e83148150b3a27076 2023-04-25 02:55:33,596 DEBUG [StoreCloser-TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/B/4f7a577a5df44d17bb388655eb7aff52 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/B/4f7a577a5df44d17bb388655eb7aff52 2023-04-25 02:55:33,597 DEBUG [StoreCloser-TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/B/886195b72dee4da182fa19a6b333d181 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/B/886195b72dee4da182fa19a6b333d181 2023-04-25 02:55:33,598 DEBUG [StoreCloser-TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/B/765c196572fd4dc9999fca242f93b77b to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/B/765c196572fd4dc9999fca242f93b77b 2023-04-25 02:55:33,598 DEBUG [StoreCloser-TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/B/3703b5da4c594ed68d58845cfd8cb0ad to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/B/3703b5da4c594ed68d58845cfd8cb0ad 2023-04-25 02:55:33,599 DEBUG [StoreCloser-TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/B/3c2de4efd527437f9f4188dc83ab23f8 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/B/3c2de4efd527437f9f4188dc83ab23f8 2023-04-25 02:55:33,600 DEBUG [StoreCloser-TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/B/f2fd638cc44b47939bbbc6e7dbea7fcd to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/B/f2fd638cc44b47939bbbc6e7dbea7fcd 2023-04-25 02:55:33,601 DEBUG [StoreCloser-TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/B/66c99592f17d4d0ca68c50099d8196ad to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/B/66c99592f17d4d0ca68c50099d8196ad 2023-04-25 02:55:33,602 DEBUG [StoreCloser-TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/B/97647b20a62d4a64aa99e34eb1fda79d to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/B/97647b20a62d4a64aa99e34eb1fda79d 2023-04-25 02:55:33,603 DEBUG [StoreCloser-TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/B/c51b3153725541ca879c6a927da493eb to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/B/c51b3153725541ca879c6a927da493eb 2023-04-25 02:55:33,604 DEBUG [StoreCloser-TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/B/2626d6f8ce5447be9decb521b7efb663 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/B/2626d6f8ce5447be9decb521b7efb663 2023-04-25 02:55:33,605 DEBUG [StoreCloser-TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/B/d317e1976c45442496ee612abf0ac5c3 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/B/d317e1976c45442496ee612abf0ac5c3 2023-04-25 02:55:33,605 DEBUG [StoreCloser-TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/B/7e0bb30ba5ae4d09b3b44e47268e1278 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/B/7e0bb30ba5ae4d09b3b44e47268e1278 2023-04-25 02:55:33,606 DEBUG [StoreCloser-TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/B/b00219107a614a578ce1ff9ef9b51c3f to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/B/b00219107a614a578ce1ff9ef9b51c3f 2023-04-25 02:55:33,607 DEBUG [StoreCloser-TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/B/66a4d84d12e24db0abb6101e481f225e to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/B/66a4d84d12e24db0abb6101e481f225e 2023-04-25 02:55:33,607 DEBUG [StoreCloser-TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/B/102dde04601f4b6491d366c3895677e0 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/B/102dde04601f4b6491d366c3895677e0 2023-04-25 02:55:33,608 DEBUG [StoreCloser-TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/B/87fec196952a43dba5e42ed48f704bb1 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/B/87fec196952a43dba5e42ed48f704bb1 2023-04-25 02:55:33,610 DEBUG [StoreCloser-TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/C/ae3ee0ab1b41449aaefda15e270fb0ae, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/C/767622ada2784abda2d8abd45232161d, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/C/f79cc538748c4dd5bed99c0e0a922ef8, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/C/73446cf519cf44218916175b681de98f, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/C/824b7a6dac994061a7b5eec7dd1c393d, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/C/68d78dd5c2834816b8ca1a20c1cbe8b4, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/C/1616975a2e1b498b82f75389ed6147a0, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/C/cc1f84c3c938484f9a6bede5da083ae5, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/C/973a1219537a4e54b56706439e8a7866, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/C/5fc6dab5c0694ae0bea08472cd197238, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/C/624fed9eeb754dfcb5b9065b5d56c272, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/C/e9523fdf37964c0ab71c8aff66e14f9b, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/C/951d1d4e165143069ac4b7d85ae7b92d, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/C/0a7842e5f47a403f9545049720c83770, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/C/4ba2891cb65b44c99fd2da78a0977bf8, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/C/67b5bda6cb9b4513ac61fd00f7b73765, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/C/ed72f11460d34a31b8dae931145a746a, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/C/123c0380fc2e44919ac8b3efc9a443bf, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/C/616aca292ee64a4cb7f7bcce4a25a9e3, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/C/a4435fdcf26140528897c769aba75e92] to archive 2023-04-25 02:55:33,611 DEBUG [StoreCloser-TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-25 02:55:33,612 DEBUG [StoreCloser-TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/C/ae3ee0ab1b41449aaefda15e270fb0ae to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/C/ae3ee0ab1b41449aaefda15e270fb0ae 2023-04-25 02:55:33,613 DEBUG [StoreCloser-TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/C/767622ada2784abda2d8abd45232161d to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/C/767622ada2784abda2d8abd45232161d 2023-04-25 02:55:33,614 DEBUG [StoreCloser-TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/C/f79cc538748c4dd5bed99c0e0a922ef8 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/C/f79cc538748c4dd5bed99c0e0a922ef8 2023-04-25 02:55:33,614 DEBUG [StoreCloser-TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/C/73446cf519cf44218916175b681de98f to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/C/73446cf519cf44218916175b681de98f 2023-04-25 02:55:33,615 DEBUG [StoreCloser-TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/C/824b7a6dac994061a7b5eec7dd1c393d to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/C/824b7a6dac994061a7b5eec7dd1c393d 2023-04-25 02:55:33,616 DEBUG [StoreCloser-TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/C/68d78dd5c2834816b8ca1a20c1cbe8b4 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/C/68d78dd5c2834816b8ca1a20c1cbe8b4 2023-04-25 02:55:33,617 DEBUG [StoreCloser-TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/C/1616975a2e1b498b82f75389ed6147a0 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/C/1616975a2e1b498b82f75389ed6147a0 2023-04-25 02:55:33,618 DEBUG [StoreCloser-TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/C/cc1f84c3c938484f9a6bede5da083ae5 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/C/cc1f84c3c938484f9a6bede5da083ae5 2023-04-25 02:55:33,618 DEBUG [StoreCloser-TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/C/973a1219537a4e54b56706439e8a7866 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/C/973a1219537a4e54b56706439e8a7866 2023-04-25 02:55:33,619 DEBUG [StoreCloser-TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/C/5fc6dab5c0694ae0bea08472cd197238 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/C/5fc6dab5c0694ae0bea08472cd197238 2023-04-25 02:55:33,620 DEBUG [StoreCloser-TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/C/624fed9eeb754dfcb5b9065b5d56c272 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/C/624fed9eeb754dfcb5b9065b5d56c272 2023-04-25 02:55:33,621 DEBUG [StoreCloser-TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/C/e9523fdf37964c0ab71c8aff66e14f9b to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/C/e9523fdf37964c0ab71c8aff66e14f9b 2023-04-25 02:55:33,621 DEBUG [StoreCloser-TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/C/951d1d4e165143069ac4b7d85ae7b92d to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/C/951d1d4e165143069ac4b7d85ae7b92d 2023-04-25 02:55:33,622 DEBUG [StoreCloser-TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/C/0a7842e5f47a403f9545049720c83770 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/C/0a7842e5f47a403f9545049720c83770 2023-04-25 02:55:33,623 DEBUG [StoreCloser-TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/C/4ba2891cb65b44c99fd2da78a0977bf8 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/C/4ba2891cb65b44c99fd2da78a0977bf8 2023-04-25 02:55:33,623 DEBUG [StoreCloser-TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/C/67b5bda6cb9b4513ac61fd00f7b73765 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/C/67b5bda6cb9b4513ac61fd00f7b73765 2023-04-25 02:55:33,624 DEBUG [StoreCloser-TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/C/ed72f11460d34a31b8dae931145a746a to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/C/ed72f11460d34a31b8dae931145a746a 2023-04-25 02:55:33,625 DEBUG [StoreCloser-TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/C/123c0380fc2e44919ac8b3efc9a443bf to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/C/123c0380fc2e44919ac8b3efc9a443bf 2023-04-25 02:55:33,625 DEBUG [StoreCloser-TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/C/616aca292ee64a4cb7f7bcce4a25a9e3 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/C/616aca292ee64a4cb7f7bcce4a25a9e3 2023-04-25 02:55:33,626 DEBUG [StoreCloser-TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/C/a4435fdcf26140528897c769aba75e92 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/C/a4435fdcf26140528897c769aba75e92 2023-04-25 02:55:33,630 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/recovered.edits/762.seqid, newMaxSeqId=762, maxSeqId=4 2023-04-25 02:55:33,631 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5. 2023-04-25 02:55:33,631 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 9d5b411b5fe67e5bb007f4d36e3e11e5: 2023-04-25 02:55:33,632 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 9d5b411b5fe67e5bb007f4d36e3e11e5 2023-04-25 02:55:33,632 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=44 updating hbase:meta row=9d5b411b5fe67e5bb007f4d36e3e11e5, regionState=CLOSED 2023-04-25 02:55:33,633 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682391333632"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682391333632"}]},"ts":"1682391333632"} 2023-04-25 02:55:33,634 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=45, resume processing ppid=44 2023-04-25 02:55:33,634 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=45, ppid=44, state=SUCCESS; CloseRegionProcedure 9d5b411b5fe67e5bb007f4d36e3e11e5, server=jenkins-hbase4.apache.org,46697,1682391220023 in 3.3170 sec 2023-04-25 02:55:33,635 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=44, resume processing ppid=43 2023-04-25 02:55:33,635 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=44, ppid=43, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=9d5b411b5fe67e5bb007f4d36e3e11e5, UNASSIGN in 3.3200 sec 2023-04-25 02:55:33,636 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682391333636"}]},"ts":"1682391333636"} 2023-04-25 02:55:33,636 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-25 02:55:33,638 INFO [PEWorker-1] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-25 02:55:33,639 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=43, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 3.3310 sec 2023-04-25 02:55:34,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-25 02:55:34,416 INFO [Listener at localhost/41915] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 43 completed 2023-04-25 02:55:34,417 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-04-25 02:55:34,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure2.ProcedureExecutor(1029): Stored pid=46, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-25 02:55:34,419 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=46, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-25 02:55:34,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1227): Checking to see if procedure is done pid=46 2023-04-25 02:55:34,419 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=46, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-25 02:55:34,424 DEBUG [HFileArchiver-9] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/.tmp/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5 2023-04-25 02:55:34,425 DEBUG [HFileArchiver-9] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/.tmp/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/A, FileablePath, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/.tmp/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/B, FileablePath, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/.tmp/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/C, FileablePath, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/.tmp/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/recovered.edits] 2023-04-25 02:55:34,429 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/.tmp/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/A/3640d17ea7ee420bb4b79e310beeb6c8 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/A/3640d17ea7ee420bb4b79e310beeb6c8 2023-04-25 02:55:34,430 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/.tmp/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/A/3e89e4af11aa40ecb9fa4f8e10d6c59e to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/A/3e89e4af11aa40ecb9fa4f8e10d6c59e 2023-04-25 02:55:34,431 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/.tmp/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/A/9209e8f26b504f3186755a8e6d2f7847 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/A/9209e8f26b504f3186755a8e6d2f7847 2023-04-25 02:55:34,432 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/.tmp/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/A/96da6605b9c9489299d34647ebed19b1 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/A/96da6605b9c9489299d34647ebed19b1 2023-04-25 02:55:34,435 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/.tmp/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/B/5ecad234252e4aa4ae9bb80cbd0f985b to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/B/5ecad234252e4aa4ae9bb80cbd0f985b 2023-04-25 02:55:34,436 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/.tmp/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/B/65409474b81645d5985504057ff9b988 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/B/65409474b81645d5985504057ff9b988 2023-04-25 02:55:34,437 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/.tmp/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/B/b9da6db969584ed5912fd6cc68b87803 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/B/b9da6db969584ed5912fd6cc68b87803 2023-04-25 02:55:34,438 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/.tmp/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/B/f9bfad94978d472e96b6a5a4b4c5b9dd to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/B/f9bfad94978d472e96b6a5a4b4c5b9dd 2023-04-25 02:55:34,440 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/.tmp/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/C/7efb1cb3804947ad8337934202335b51 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/C/7efb1cb3804947ad8337934202335b51 2023-04-25 02:55:34,441 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/.tmp/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/C/9bafd9e804ff451188b2045fc8468d49 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/C/9bafd9e804ff451188b2045fc8468d49 2023-04-25 02:55:34,442 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/.tmp/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/C/a21351296fe64c72b088bc29363c0672 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/C/a21351296fe64c72b088bc29363c0672 2023-04-25 02:55:34,443 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/.tmp/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/C/dbccc78f5b4f406eb56b73e4ce34c9c2 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/C/dbccc78f5b4f406eb56b73e4ce34c9c2 2023-04-25 02:55:34,445 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/.tmp/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/recovered.edits/762.seqid to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5/recovered.edits/762.seqid 2023-04-25 02:55:34,446 DEBUG [HFileArchiver-9] backup.HFileArchiver(596): Deleted hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/.tmp/data/default/TestAcidGuarantees/9d5b411b5fe67e5bb007f4d36e3e11e5 2023-04-25 02:55:34,446 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-25 02:55:34,446 DEBUG [PEWorker-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-25 02:55:34,447 DEBUG [PEWorker-5] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A] 2023-04-25 02:55:34,450 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425192ab37a9d824ced91e05413c8a1fef6 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425192ab37a9d824ced91e05413c8a1fef6 2023-04-25 02:55:34,451 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042523ba0fe2ca7448899fe9db2e7457bffa to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042523ba0fe2ca7448899fe9db2e7457bffa 2023-04-25 02:55:34,452 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304254255a15a98ea4b90a77d4646a65cb9db to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304254255a15a98ea4b90a77d4646a65cb9db 2023-04-25 02:55:34,454 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304254a475f44d20547209efa13bd2c9aa83c to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304254a475f44d20547209efa13bd2c9aa83c 2023-04-25 02:55:34,455 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304254a851860ab344ca49f9d35382604779a to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304254a851860ab344ca49f9d35382604779a 2023-04-25 02:55:34,456 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304256336f64ccdad40c48647833b279fc760 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304256336f64ccdad40c48647833b279fc760 2023-04-25 02:55:34,457 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042567177bf144144e7c87d64c73089449eb to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042567177bf144144e7c87d64c73089449eb 2023-04-25 02:55:34,458 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425697234a67fd644a7b97272ebc1a5839d to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425697234a67fd644a7b97272ebc1a5839d 2023-04-25 02:55:34,459 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304257fd2ca48b8c046c1a27937db09ade548 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304257fd2ca48b8c046c1a27937db09ade548 2023-04-25 02:55:34,461 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425879da4c53fb34c5b97aa7bdbadeed2cb to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425879da4c53fb34c5b97aa7bdbadeed2cb 2023-04-25 02:55:34,462 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304258ba9b8c9f4574eee9b96607ef8a8e6b6 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304258ba9b8c9f4574eee9b96607ef8a8e6b6 2023-04-25 02:55:34,463 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304259282819193c44462b0130d7f74931c0f to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304259282819193c44462b0130d7f74931c0f 2023-04-25 02:55:34,464 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425a73d2819ec5f48a2ab99fc5fea77d477 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425a73d2819ec5f48a2ab99fc5fea77d477 2023-04-25 02:55:34,465 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425bbc6ab684edc4a4eb8032431c1832142 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425bbc6ab684edc4a4eb8032431c1832142 2023-04-25 02:55:34,466 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425be9013c86466429c924b2088c228e97c to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425be9013c86466429c924b2088c228e97c 2023-04-25 02:55:34,467 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425c53e91b5c1684163ba26c6914d561ff0 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425c53e91b5c1684163ba26c6914d561ff0 2023-04-25 02:55:34,468 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425ca0eb20bef6c40e0972fe84fcdbb70c1 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425ca0eb20bef6c40e0972fe84fcdbb70c1 2023-04-25 02:55:34,469 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425e2dfced89dbc43cb87fc6f5c7bcb7643 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425e2dfced89dbc43cb87fc6f5c7bcb7643 2023-04-25 02:55:34,470 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425e4497fccd6e84b449d86d1b144cd9bbe to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425e4497fccd6e84b449d86d1b144cd9bbe 2023-04-25 02:55:34,471 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425e6a540a70d9f4bfa92b4d436004f167f to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425e6a540a70d9f4bfa92b4d436004f167f 2023-04-25 02:55:34,472 DEBUG [PEWorker-5] backup.HFileArchiver(596): Deleted hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-25 02:55:34,474 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=46, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-25 02:55:34,475 WARN [PEWorker-5] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-25 02:55:34,476 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-25 02:55:34,477 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=46, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-25 02:55:34,477 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-25 02:55:34,477 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1682391334477"}]},"ts":"9223372036854775807"} 2023-04-25 02:55:34,479 INFO [PEWorker-5] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-25 02:55:34,479 DEBUG [PEWorker-5] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 9d5b411b5fe67e5bb007f4d36e3e11e5, NAME => 'TestAcidGuarantees,,1682391301831.9d5b411b5fe67e5bb007f4d36e3e11e5.', STARTKEY => '', ENDKEY => ''}] 2023-04-25 02:55:34,479 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-25 02:55:34,479 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1682391334479"}]},"ts":"9223372036854775807"} 2023-04-25 02:55:34,480 INFO [PEWorker-5] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-25 02:55:34,482 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(130): Finished pid=46, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-25 02:55:34,483 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=46, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 65 msec 2023-04-25 02:55:34,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1227): Checking to see if procedure is done pid=46 2023-04-25 02:55:34,520 INFO [Listener at localhost/41915] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 46 completed 2023-04-25 02:55:34,531 INFO [Listener at localhost/41915] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMobScanAtomicity Thread=225 (was 229), OpenFileDescriptor=528 (was 576), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=418 (was 445), ProcessCount=170 (was 170), AvailableMemoryMB=4191 (was 4257) 2023-04-25 02:55:34,542 INFO [Listener at localhost/41915] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testScanAtomicity Thread=225, OpenFileDescriptor=528, MaxFileDescriptor=60000, SystemLoadAverage=418, ProcessCount=170, AvailableMemoryMB=4191 2023-04-25 02:55:34,544 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-25 02:55:34,544 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-25 02:55:34,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure2.ProcedureExecutor(1029): Stored pid=47, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-25 02:55:34,546 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-25 02:55:34,547 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 47 2023-04-25 02:55:34,547 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-25 02:55:34,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-25 02:55:34,548 DEBUG [HFileArchiver-10] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/.tmp/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c 2023-04-25 02:55:34,549 DEBUG [HFileArchiver-10] backup.HFileArchiver(153): Directory hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/.tmp/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c empty. 2023-04-25 02:55:34,549 DEBUG [HFileArchiver-10] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/.tmp/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c 2023-04-25 02:55:34,549 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-25 02:55:34,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-25 02:55:34,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-25 02:55:34,956 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-25 02:55:34,958 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 5f2c97e00c770d329321cff2fbd6848c, NAME => 'TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/.tmp 2023-04-25 02:55:35,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-25 02:55:35,365 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-25 02:55:35,365 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 5f2c97e00c770d329321cff2fbd6848c, disabling compactions & flushes 2023-04-25 02:55:35,365 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. 2023-04-25 02:55:35,365 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. 2023-04-25 02:55:35,365 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. after waiting 0 ms 2023-04-25 02:55:35,365 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. 2023-04-25 02:55:35,365 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. 2023-04-25 02:55:35,365 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 5f2c97e00c770d329321cff2fbd6848c: 2023-04-25 02:55:35,367 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-25 02:55:35,368 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682391335368"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682391335368"}]},"ts":"1682391335368"} 2023-04-25 02:55:35,369 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-25 02:55:35,370 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-25 02:55:35,370 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682391335370"}]},"ts":"1682391335370"} 2023-04-25 02:55:35,370 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-25 02:55:35,377 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=48, ppid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=5f2c97e00c770d329321cff2fbd6848c, ASSIGN}] 2023-04-25 02:55:35,378 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=48, ppid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=5f2c97e00c770d329321cff2fbd6848c, ASSIGN 2023-04-25 02:55:35,378 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=48, ppid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=5f2c97e00c770d329321cff2fbd6848c, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,46697,1682391220023; forceNewPlan=false, retain=false 2023-04-25 02:55:35,530 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=48 updating hbase:meta row=5f2c97e00c770d329321cff2fbd6848c, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:35,530 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682391335530"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682391335530"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682391335530"}]},"ts":"1682391335530"} 2023-04-25 02:55:35,531 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=49, ppid=48, state=RUNNABLE; OpenRegionProcedure 5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023}] 2023-04-25 02:55:35,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-25 02:55:35,686 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. 2023-04-25 02:55:35,686 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 5f2c97e00c770d329321cff2fbd6848c, NAME => 'TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c.', STARTKEY => '', ENDKEY => ''} 2023-04-25 02:55:35,686 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 5f2c97e00c770d329321cff2fbd6848c 2023-04-25 02:55:35,686 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-25 02:55:35,686 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 5f2c97e00c770d329321cff2fbd6848c 2023-04-25 02:55:35,686 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 5f2c97e00c770d329321cff2fbd6848c 2023-04-25 02:55:35,687 INFO [StoreOpener-5f2c97e00c770d329321cff2fbd6848c-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 5f2c97e00c770d329321cff2fbd6848c 2023-04-25 02:55:35,689 DEBUG [StoreOpener-5f2c97e00c770d329321cff2fbd6848c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A 2023-04-25 02:55:35,689 DEBUG [StoreOpener-5f2c97e00c770d329321cff2fbd6848c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A 2023-04-25 02:55:35,689 INFO [StoreOpener-5f2c97e00c770d329321cff2fbd6848c-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-25 02:55:35,689 INFO [StoreOpener-5f2c97e00c770d329321cff2fbd6848c-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 5f2c97e00c770d329321cff2fbd6848c columnFamilyName A 2023-04-25 02:55:35,689 INFO [StoreOpener-5f2c97e00c770d329321cff2fbd6848c-1] regionserver.HStore(310): Store=5f2c97e00c770d329321cff2fbd6848c/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-25 02:55:35,690 INFO [StoreOpener-5f2c97e00c770d329321cff2fbd6848c-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 5f2c97e00c770d329321cff2fbd6848c 2023-04-25 02:55:35,691 DEBUG [StoreOpener-5f2c97e00c770d329321cff2fbd6848c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B 2023-04-25 02:55:35,691 DEBUG [StoreOpener-5f2c97e00c770d329321cff2fbd6848c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B 2023-04-25 02:55:35,691 INFO [StoreOpener-5f2c97e00c770d329321cff2fbd6848c-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-25 02:55:35,691 INFO [StoreOpener-5f2c97e00c770d329321cff2fbd6848c-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 5f2c97e00c770d329321cff2fbd6848c columnFamilyName B 2023-04-25 02:55:35,691 INFO [StoreOpener-5f2c97e00c770d329321cff2fbd6848c-1] regionserver.HStore(310): Store=5f2c97e00c770d329321cff2fbd6848c/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-25 02:55:35,691 INFO [StoreOpener-5f2c97e00c770d329321cff2fbd6848c-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 5f2c97e00c770d329321cff2fbd6848c 2023-04-25 02:55:35,692 DEBUG [StoreOpener-5f2c97e00c770d329321cff2fbd6848c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C 2023-04-25 02:55:35,692 DEBUG [StoreOpener-5f2c97e00c770d329321cff2fbd6848c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C 2023-04-25 02:55:35,693 INFO [StoreOpener-5f2c97e00c770d329321cff2fbd6848c-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-25 02:55:35,693 INFO [StoreOpener-5f2c97e00c770d329321cff2fbd6848c-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 5f2c97e00c770d329321cff2fbd6848c columnFamilyName C 2023-04-25 02:55:35,693 INFO [StoreOpener-5f2c97e00c770d329321cff2fbd6848c-1] regionserver.HStore(310): Store=5f2c97e00c770d329321cff2fbd6848c/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-25 02:55:35,693 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. 2023-04-25 02:55:35,694 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c 2023-04-25 02:55:35,694 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c 2023-04-25 02:55:35,696 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-25 02:55:35,697 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 5f2c97e00c770d329321cff2fbd6848c 2023-04-25 02:55:35,698 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-25 02:55:35,699 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 5f2c97e00c770d329321cff2fbd6848c; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=71511185, jitterRate=0.06559969484806061}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-25 02:55:35,699 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 5f2c97e00c770d329321cff2fbd6848c: 2023-04-25 02:55:35,699 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c., pid=49, masterSystemTime=1682391335683 2023-04-25 02:55:35,700 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. 2023-04-25 02:55:35,700 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. 2023-04-25 02:55:35,701 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=48 updating hbase:meta row=5f2c97e00c770d329321cff2fbd6848c, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:35,701 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682391335701"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1682391335701"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1682391335701"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1682391335701"}]},"ts":"1682391335701"} 2023-04-25 02:55:35,703 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=49, resume processing ppid=48 2023-04-25 02:55:35,703 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=49, ppid=48, state=SUCCESS; OpenRegionProcedure 5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 in 171 msec 2023-04-25 02:55:35,704 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=48, resume processing ppid=47 2023-04-25 02:55:35,704 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=48, ppid=47, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=5f2c97e00c770d329321cff2fbd6848c, ASSIGN in 327 msec 2023-04-25 02:55:35,704 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-25 02:55:35,704 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682391335704"}]},"ts":"1682391335704"} 2023-04-25 02:55:35,705 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-25 02:55:35,707 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-25 02:55:35,707 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=47, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1620 sec 2023-04-25 02:55:36,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-25 02:55:36,652 INFO [Listener at localhost/41915] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 47 completed 2023-04-25 02:55:36,652 DEBUG [Listener at localhost/41915] zookeeper.ReadOnlyZKClient(139): Connect 0x20ed9117 to 127.0.0.1:62054 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-25 02:55:36,658 DEBUG [Listener at localhost/41915] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@34ad4aa8, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-25 02:55:36,659 DEBUG [Listener at localhost/41915] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-25 02:55:36,662 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59102, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-25 02:55:36,662 DEBUG [Listener at localhost/41915] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-25 02:55:36,664 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:38322, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-25 02:55:36,665 DEBUG [Listener at localhost/41915] zookeeper.ReadOnlyZKClient(139): Connect 0x50002343 to 127.0.0.1:62054 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-25 02:55:36,669 DEBUG [Listener at localhost/41915] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1c29757d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-25 02:55:36,669 DEBUG [Listener at localhost/41915] zookeeper.ReadOnlyZKClient(139): Connect 0x5726fb8b to 127.0.0.1:62054 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-25 02:55:36,683 DEBUG [Listener at localhost/41915] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6edd017, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-25 02:55:36,683 DEBUG [Listener at localhost/41915] zookeeper.ReadOnlyZKClient(139): Connect 0x6f2b3c03 to 127.0.0.1:62054 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-25 02:55:36,699 DEBUG [Listener at localhost/41915] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@fb168ce, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-25 02:55:36,699 DEBUG [Listener at localhost/41915] zookeeper.ReadOnlyZKClient(139): Connect 0x70e2be04 to 127.0.0.1:62054 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-25 02:55:36,715 DEBUG [Listener at localhost/41915] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@461f5f3f, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-25 02:55:36,715 DEBUG [Listener at localhost/41915] zookeeper.ReadOnlyZKClient(139): Connect 0x067ea274 to 127.0.0.1:62054 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-25 02:55:36,721 DEBUG [Listener at localhost/41915] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@63093e21, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-25 02:55:36,722 DEBUG [Listener at localhost/41915] zookeeper.ReadOnlyZKClient(139): Connect 0x0b475d8d to 127.0.0.1:62054 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-25 02:55:36,737 DEBUG [Listener at localhost/41915] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@211582f2, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-25 02:55:36,737 DEBUG [Listener at localhost/41915] zookeeper.ReadOnlyZKClient(139): Connect 0x549bef45 to 127.0.0.1:62054 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-25 02:55:36,752 DEBUG [Listener at localhost/41915] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@65816b6e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-25 02:55:36,753 DEBUG [Listener at localhost/41915] zookeeper.ReadOnlyZKClient(139): Connect 0x6ed183a4 to 127.0.0.1:62054 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-25 02:55:36,762 DEBUG [Listener at localhost/41915] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@653bb8fc, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-25 02:55:36,762 DEBUG [Listener at localhost/41915] zookeeper.ReadOnlyZKClient(139): Connect 0x10a344e6 to 127.0.0.1:62054 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-25 02:55:36,767 DEBUG [Listener at localhost/41915] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@720d29f5, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-25 02:55:36,768 DEBUG [Listener at localhost/41915] zookeeper.ReadOnlyZKClient(139): Connect 0x3a9f3972 to 127.0.0.1:62054 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-25 02:55:36,775 DEBUG [Listener at localhost/41915] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3011deb3, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-25 02:55:36,779 DEBUG [hconnection-0x32a120fb-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-25 02:55:36,779 DEBUG [hconnection-0x71bba646-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-25 02:55:36,780 DEBUG [hconnection-0x208161a2-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-25 02:55:36,780 DEBUG [hconnection-0x1939f8d8-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-25 02:55:36,780 DEBUG [hconnection-0x3d3e9cad-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-25 02:55:36,780 DEBUG [hconnection-0x1b7cf799-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-25 02:55:36,780 DEBUG [hconnection-0x4190b8af-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-25 02:55:36,785 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:55:36,785 DEBUG [hconnection-0x4bd1e522-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-25 02:55:36,785 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59114, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-25 02:55:36,785 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59122, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-25 02:55:36,786 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59134, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-25 02:55:36,786 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59136, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-25 02:55:36,787 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59156, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-25 02:55:36,787 DEBUG [hconnection-0x1b921657-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-25 02:55:36,787 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59116, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-25 02:55:36,791 DEBUG [hconnection-0x567aa66f-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-25 02:55:36,794 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59126, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-25 02:55:36,795 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59144, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-25 02:55:36,795 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59162, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-25 02:55:36,797 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59176, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-25 02:55:36,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:55:36,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:55:36,799 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:55:36,799 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:36,799 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:55:36,799 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:55:36,800 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:36,800 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:36,802 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:36,802 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:36,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:36,802 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:36,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:36,802 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:36,803 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:55:36,803 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:36,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:55:36,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:36,803 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:55:36,804 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:55:36,804 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:55:36,804 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:55:36,804 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:36,804 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:55:36,804 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:55:36,804 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:55:36,804 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. 2023-04-25 02:55:36,805 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. started... 2023-04-25 02:55:36,805 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5f2c97e00c770d329321cff2fbd6848c 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-04-25 02:55:36,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(9158): Flush requested on 5f2c97e00c770d329321cff2fbd6848c 2023-04-25 02:55:36,809 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. as already flushing 2023-04-25 02:55:36,809 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5f2c97e00c770d329321cff2fbd6848c, store=A 2023-04-25 02:55:36,809 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:55:36,809 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5f2c97e00c770d329321cff2fbd6848c, store=B 2023-04-25 02:55:36,809 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:55:36,809 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5f2c97e00c770d329321cff2fbd6848c, store=C 2023-04-25 02:55:36,809 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:55:36,821 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682391336818, max timestamp=1682391336819Num uniques 859; 2023-04-25 02:55:36,821 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1682391336818, max timestamp=1682391336819Num uniques 859; 2023-04-25 02:55:36,821 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-25 02:55:36,821 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-25 02:55:36,821 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1682391336818, max timestamp=1682391336819Num uniques 859; 2023-04-25 02:55:36,821 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-25 02:55:36,840 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:36,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59136 deadline: 1682391396834, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:36,840 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:36,841 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59162 deadline: 1682391396835, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:36,841 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:36,841 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59116 deadline: 1682391396835, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:36,841 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:36,841 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59114 deadline: 1682391396838, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:36,844 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:36,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59156 deadline: 1682391396840, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:36,945 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:36,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59136 deadline: 1682391396941, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:36,945 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:36,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59162 deadline: 1682391396941, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:36,945 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:36,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59116 deadline: 1682391396942, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:36,945 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:36,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59114 deadline: 1682391396942, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:36,949 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:36,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59156 deadline: 1682391396945, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:37,151 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:37,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59116 deadline: 1682391397146, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:37,152 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:37,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59114 deadline: 1682391397146, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:37,152 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:37,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59136 deadline: 1682391397147, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:37,152 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:37,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59162 deadline: 1682391397147, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:37,154 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:37,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59156 deadline: 1682391397150, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:37,227 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=12 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp/A/978bb4c158c9443b964b311f5b9fdcb9 2023-04-25 02:55:37,456 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:37,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59136 deadline: 1682391397453, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:37,459 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:37,459 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59162 deadline: 1682391397454, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:37,459 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:37,459 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59116 deadline: 1682391397455, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:37,459 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:37,459 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59156 deadline: 1682391397455, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:37,459 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:37,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59114 deadline: 1682391397455, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:37,638 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=12 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp/B/2558ca7b08414e49a9823e8510998a7e 2023-04-25 02:55:37,648 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=12 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp/C/90075ab23b324726b3123f4930955837 2023-04-25 02:55:37,652 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp/A/978bb4c158c9443b964b311f5b9fdcb9 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/978bb4c158c9443b964b311f5b9fdcb9 2023-04-25 02:55:37,664 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/978bb4c158c9443b964b311f5b9fdcb9, entries=150, sequenceid=12, filesize=11.6 K 2023-04-25 02:55:37,665 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp/B/2558ca7b08414e49a9823e8510998a7e as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/2558ca7b08414e49a9823e8510998a7e 2023-04-25 02:55:37,672 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/2558ca7b08414e49a9823e8510998a7e, entries=150, sequenceid=12, filesize=11.6 K 2023-04-25 02:55:37,673 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp/C/90075ab23b324726b3123f4930955837 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/90075ab23b324726b3123f4930955837 2023-04-25 02:55:37,682 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/90075ab23b324726b3123f4930955837, entries=150, sequenceid=12, filesize=11.6 K 2023-04-25 02:55:37,682 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 5f2c97e00c770d329321cff2fbd6848c in 877ms, sequenceid=12, compaction requested=false 2023-04-25 02:55:37,683 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5f2c97e00c770d329321cff2fbd6848c: 2023-04-25 02:55:37,683 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. 2023-04-25 02:55:37,683 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:55:37,683 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:55:37,683 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:37,683 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:55:37,683 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:55:37,685 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:37,685 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:37,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:37,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:37,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:37,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:37,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:37,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:37,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:37,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:37,687 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:55:37,687 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@35bba28[Count = 0] remaining members to acquire global barrier 2023-04-25 02:55:37,687 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:55:37,687 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:37,687 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:37,687 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:55:37,688 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:37,688 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:37,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:37,688 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:55:37,688 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:55:37,688 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:55:37,688 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:37,689 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:55:37,690 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:37,690 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:55:37,690 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:37,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:37,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:37,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:37,690 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:55:37,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:37,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:37,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:37,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:37,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:37,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:37,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:37,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:55:37,693 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:55:37,693 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:55:37,693 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:55:37,693 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:55:37,694 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:55:37,694 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:55:37,695 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:37,695 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:37,695 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:37,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:37,695 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:37,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:37,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:37,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:37,696 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:37,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:37,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:37,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:37,697 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:37,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:37,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:37,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:37,698 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:37,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:37,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:37,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:37,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:37,702 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:37,702 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:37,702 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:37,702 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:37,702 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:37,702 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:55:37,702 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:55:37,702 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:37,702 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:37,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:37,702 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:37,702 DEBUG [Thread-2900] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:55:37,702 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:37,702 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:55:37,703 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:37,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:37,703 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:37,703 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:37,703 DEBUG [Thread-2900] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:55:37,803 DEBUG [Thread-2900] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:55:37,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:55:37,805 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:55:37,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:55:37,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:55:37,808 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:55:37,808 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:37,808 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:55:37,808 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:55:37,808 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:37,809 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:37,810 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:37,810 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:37,810 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:37,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:37,810 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:37,810 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:55:37,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:37,810 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:37,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:55:37,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:37,811 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:55:37,811 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:55:37,811 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:55:37,811 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:55:37,811 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:37,811 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:55:37,811 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:55:37,811 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:55:37,811 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. 2023-04-25 02:55:37,811 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. started... 2023-04-25 02:55:37,812 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5f2c97e00c770d329321cff2fbd6848c 3/3 column families, dataSize=127.47 KB heapSize=360.38 KB 2023-04-25 02:55:37,812 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5f2c97e00c770d329321cff2fbd6848c, store=A 2023-04-25 02:55:37,812 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 02:55:37,812 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5f2c97e00c770d329321cff2fbd6848c, store=B 2023-04-25 02:55:37,812 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 02:55:37,812 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5f2c97e00c770d329321cff2fbd6848c, store=C 2023-04-25 02:55:37,812 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 02:55:37,822 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=47 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp/A/8c4835a8c95c4d9eacfb721cd6590da2 2023-04-25 02:55:37,836 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=47 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp/B/69850179080a482fb3fe733a789862c5 2023-04-25 02:55:37,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(9158): Flush requested on 5f2c97e00c770d329321cff2fbd6848c 2023-04-25 02:55:37,968 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. as already flushing 2023-04-25 02:55:37,981 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:37,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59116 deadline: 1682391397971, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:37,988 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:37,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59114 deadline: 1682391397979, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:37,991 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:37,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59136 deadline: 1682391397980, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:37,992 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:37,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59162 deadline: 1682391397981, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:37,992 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:37,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59156 deadline: 1682391397981, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:38,086 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:38,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59116 deadline: 1682391398082, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:38,095 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:38,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59114 deadline: 1682391398090, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:38,097 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:38,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59136 deadline: 1682391398092, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:38,098 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:38,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59162 deadline: 1682391398094, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:38,101 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:38,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59156 deadline: 1682391398096, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:38,247 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=47 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp/C/1846ea21c79a487f821d3bbecf00b000 2023-04-25 02:55:38,252 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp/A/8c4835a8c95c4d9eacfb721cd6590da2 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/8c4835a8c95c4d9eacfb721cd6590da2 2023-04-25 02:55:38,255 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/8c4835a8c95c4d9eacfb721cd6590da2, entries=150, sequenceid=47, filesize=11.6 K 2023-04-25 02:55:38,256 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp/B/69850179080a482fb3fe733a789862c5 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/69850179080a482fb3fe733a789862c5 2023-04-25 02:55:38,259 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/69850179080a482fb3fe733a789862c5, entries=150, sequenceid=47, filesize=11.6 K 2023-04-25 02:55:38,260 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp/C/1846ea21c79a487f821d3bbecf00b000 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/1846ea21c79a487f821d3bbecf00b000 2023-04-25 02:55:38,263 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/1846ea21c79a487f821d3bbecf00b000, entries=150, sequenceid=47, filesize=11.6 K 2023-04-25 02:55:38,263 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~360.33 KB/368976, currentSize=60.38 KB/61830 for 5f2c97e00c770d329321cff2fbd6848c in 451ms, sequenceid=47, compaction requested=false 2023-04-25 02:55:38,263 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5f2c97e00c770d329321cff2fbd6848c: 2023-04-25 02:55:38,263 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. 2023-04-25 02:55:38,263 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:55:38,264 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:55:38,264 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:38,264 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:55:38,264 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:55:38,269 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:38,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:38,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:38,269 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:38,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:38,270 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:38,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:38,270 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:55:38,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:38,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:38,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:38,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:38,271 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:55:38,271 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4266e44a[Count = 0] remaining members to acquire global barrier 2023-04-25 02:55:38,271 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:55:38,271 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:38,272 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:38,272 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:38,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:38,272 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:55:38,272 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:55:38,272 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:55:38,275 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:55:38,275 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:38,275 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:38,275 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:55:38,275 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:55:38,275 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:55:38,275 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:55:38,276 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:55:38,276 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:55:38,276 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:55:38,276 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:55:38,277 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:38,277 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:38,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:38,277 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:38,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:38,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:38,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:38,277 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:38,278 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:38,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:38,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:38,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:38,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:38,279 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:38,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:38,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:38,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:38,281 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:38,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:38,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:38,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:38,283 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:38,283 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:38,283 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:38,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:38,283 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:38,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:38,284 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:38,284 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:55:38,285 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:55:38,284 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:38,285 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:38,285 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:38,285 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:38,285 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:38,285 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:38,285 DEBUG [Thread-2900] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:55:38,285 DEBUG [Thread-2900] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:55:38,285 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:38,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(9158): Flush requested on 5f2c97e00c770d329321cff2fbd6848c 2023-04-25 02:55:38,292 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 5f2c97e00c770d329321cff2fbd6848c 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-25 02:55:38,293 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5f2c97e00c770d329321cff2fbd6848c, store=A 2023-04-25 02:55:38,293 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:55:38,293 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5f2c97e00c770d329321cff2fbd6848c, store=B 2023-04-25 02:55:38,293 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:55:38,293 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5f2c97e00c770d329321cff2fbd6848c, store=C 2023-04-25 02:55:38,293 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:55:38,322 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=60 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp/A/9752f6e742dd42a695156126d7f9adb9 2023-04-25 02:55:38,330 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:38,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59114 deadline: 1682391398322, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:38,330 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:38,331 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59116 deadline: 1682391398323, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:38,335 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:38,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59162 deadline: 1682391398325, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:38,335 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:38,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59136 deadline: 1682391398325, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:38,335 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:38,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59156 deadline: 1682391398325, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:38,345 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=60 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp/B/40fbc1e0fc064f6283aa283f185ceb0d 2023-04-25 02:55:38,377 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=60 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp/C/fb6b837b3def4a1c88bbb59f078426d6 2023-04-25 02:55:38,382 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp/A/9752f6e742dd42a695156126d7f9adb9 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/9752f6e742dd42a695156126d7f9adb9 2023-04-25 02:55:38,385 DEBUG [Thread-2900] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:55:38,385 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/9752f6e742dd42a695156126d7f9adb9, entries=200, sequenceid=60, filesize=13.9 K 2023-04-25 02:55:38,386 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:55:38,386 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp/B/40fbc1e0fc064f6283aa283f185ceb0d as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/40fbc1e0fc064f6283aa283f185ceb0d 2023-04-25 02:55:38,387 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:55:38,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:55:38,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:55:38,389 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:55:38,389 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:38,389 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:55:38,389 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:55:38,389 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:38,389 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:38,390 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:38,390 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:38,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:38,390 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:38,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:38,391 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:38,391 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:55:38,391 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:38,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:55:38,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:38,391 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:55:38,392 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:55:38,392 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:55:38,392 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:55:38,392 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:38,392 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:55:38,393 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/40fbc1e0fc064f6283aa283f185ceb0d, entries=150, sequenceid=60, filesize=11.6 K 2023-04-25 02:55:38,394 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:55:38,394 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:55:38,394 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. 2023-04-25 02:55:38,394 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. started... 2023-04-25 02:55:38,394 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp/C/fb6b837b3def4a1c88bbb59f078426d6 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/fb6b837b3def4a1c88bbb59f078426d6 2023-04-25 02:55:38,394 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. as already flushing 2023-04-25 02:55:38,394 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. 2023-04-25 02:55:38,394 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:55:38,394 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:55:38,394 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:38,394 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:55:38,394 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:55:38,402 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:38,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:38,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:38,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:38,402 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:38,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:38,402 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:38,402 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:55:38,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:38,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:38,403 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/fb6b837b3def4a1c88bbb59f078426d6, entries=150, sequenceid=60, filesize=11.6 K 2023-04-25 02:55:38,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:38,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:38,405 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 5f2c97e00c770d329321cff2fbd6848c in 113ms, sequenceid=60, compaction requested=true 2023-04-25 02:55:38,405 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 5f2c97e00c770d329321cff2fbd6848c: 2023-04-25 02:55:38,405 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:55:38,405 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 02:55:38,405 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-25 02:55:38,405 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-25 02:55:38,405 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 02:55:38,405 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:55:38,405 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:38,405 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@81a6dbf[Count = 0] remaining members to acquire global barrier 2023-04-25 02:55:38,405 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 02:55:38,406 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37998 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-25 02:55:38,406 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-25 02:55:38,406 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HStore(1912): 5f2c97e00c770d329321cff2fbd6848c/A is initiating minor compaction (all files) 2023-04-25 02:55:38,406 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HStore(1912): 5f2c97e00c770d329321cff2fbd6848c/B is initiating minor compaction (all files) 2023-04-25 02:55:38,406 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 5f2c97e00c770d329321cff2fbd6848c/A in TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. 2023-04-25 02:55:38,406 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 5f2c97e00c770d329321cff2fbd6848c/B in TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. 2023-04-25 02:55:38,407 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/978bb4c158c9443b964b311f5b9fdcb9, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/8c4835a8c95c4d9eacfb721cd6590da2, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/9752f6e742dd42a695156126d7f9adb9] into tmpdir=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp, totalSize=37.1 K 2023-04-25 02:55:38,407 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/2558ca7b08414e49a9823e8510998a7e, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/69850179080a482fb3fe733a789862c5, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/40fbc1e0fc064f6283aa283f185ceb0d] into tmpdir=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp, totalSize=34.8 K 2023-04-25 02:55:38,407 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:38,407 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:55:38,407 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting 978bb4c158c9443b964b311f5b9fdcb9, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=12, earliestPutTs=1682391336800 2023-04-25 02:55:38,407 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:38,408 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting 2558ca7b08414e49a9823e8510998a7e, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=12, earliestPutTs=1682391336800 2023-04-25 02:55:38,408 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:38,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:38,408 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:55:38,408 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:55:38,408 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:55:38,408 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting 69850179080a482fb3fe733a789862c5, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=47, earliestPutTs=1682391336832 2023-04-25 02:55:38,408 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting 8c4835a8c95c4d9eacfb721cd6590da2, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=47, earliestPutTs=1682391336832 2023-04-25 02:55:38,408 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting 40fbc1e0fc064f6283aa283f185ceb0d, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=60, earliestPutTs=1682391337978 2023-04-25 02:55:38,409 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting 9752f6e742dd42a695156126d7f9adb9, keycount=200, bloomtype=ROW, size=13.9 K, encoding=NONE, compression=NONE, seqNum=60, earliestPutTs=1682391337976 2023-04-25 02:55:38,410 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:38,410 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:55:38,410 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:38,410 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:55:38,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:38,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:38,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:38,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:38,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:38,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:38,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:38,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:38,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:38,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:38,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:55:38,414 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:55:38,414 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:55:38,414 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:55:38,414 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:55:38,414 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:55:38,414 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:55:38,418 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] throttle.PressureAwareThroughputController(145): 5f2c97e00c770d329321cff2fbd6848c#B#compaction#375 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 53.85 MB/second 2023-04-25 02:55:38,418 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] throttle.PressureAwareThroughputController(145): 5f2c97e00c770d329321cff2fbd6848c#A#compaction#376 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-25 02:55:38,419 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:38,419 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:38,419 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:38,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:38,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:38,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:38,420 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:38,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:38,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:38,420 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:38,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:38,421 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:38,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:38,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:38,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:38,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:38,422 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:38,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:38,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:38,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:38,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:38,431 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:38,431 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:38,431 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:38,431 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:55:38,431 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:55:38,431 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:38,431 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:38,431 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:38,431 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:38,431 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:38,431 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:38,431 DEBUG [Thread-2900] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:55:38,431 DEBUG [Thread-2900] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:55:38,431 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:38,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:38,431 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:38,432 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:38,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:38,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(9158): Flush requested on 5f2c97e00c770d329321cff2fbd6848c 2023-04-25 02:55:38,441 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 5f2c97e00c770d329321cff2fbd6848c 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-25 02:55:38,442 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5f2c97e00c770d329321cff2fbd6848c, store=A 2023-04-25 02:55:38,442 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:55:38,442 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5f2c97e00c770d329321cff2fbd6848c, store=B 2023-04-25 02:55:38,442 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:55:38,442 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5f2c97e00c770d329321cff2fbd6848c, store=C 2023-04-25 02:55:38,442 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:55:38,442 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp/B/1fc484b1f6d44b9aa4d98ea4c00115bc as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/1fc484b1f6d44b9aa4d98ea4c00115bc 2023-04-25 02:55:38,448 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 5f2c97e00c770d329321cff2fbd6848c/B of 5f2c97e00c770d329321cff2fbd6848c into 1fc484b1f6d44b9aa4d98ea4c00115bc(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 02:55:38,448 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5f2c97e00c770d329321cff2fbd6848c: 2023-04-25 02:55:38,448 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c., storeName=5f2c97e00c770d329321cff2fbd6848c/B, priority=13, startTime=1682391338405; duration=0sec 2023-04-25 02:55:38,448 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 02:55:38,448 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-25 02:55:38,449 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-25 02:55:38,449 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HStore(1912): 5f2c97e00c770d329321cff2fbd6848c/C is initiating minor compaction (all files) 2023-04-25 02:55:38,449 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 5f2c97e00c770d329321cff2fbd6848c/C in TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. 2023-04-25 02:55:38,449 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/90075ab23b324726b3123f4930955837, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/1846ea21c79a487f821d3bbecf00b000, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/fb6b837b3def4a1c88bbb59f078426d6] into tmpdir=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp, totalSize=34.8 K 2023-04-25 02:55:38,449 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting 90075ab23b324726b3123f4930955837, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=12, earliestPutTs=1682391336800 2023-04-25 02:55:38,450 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting 1846ea21c79a487f821d3bbecf00b000, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=47, earliestPutTs=1682391336832 2023-04-25 02:55:38,450 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting fb6b837b3def4a1c88bbb59f078426d6, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=60, earliestPutTs=1682391337978 2023-04-25 02:55:38,462 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:38,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59114 deadline: 1682391398452, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:38,462 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:38,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59116 deadline: 1682391398453, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:38,463 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:38,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59162 deadline: 1682391398454, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:38,464 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] throttle.PressureAwareThroughputController(145): 5f2c97e00c770d329321cff2fbd6848c#C#compaction#378 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-25 02:55:38,466 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:38,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59156 deadline: 1682391398458, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:38,466 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:38,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59136 deadline: 1682391398463, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:38,483 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp/C/c39af132f78942f5a05423334d8ab1b6 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/c39af132f78942f5a05423334d8ab1b6 2023-04-25 02:55:38,496 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 5f2c97e00c770d329321cff2fbd6848c/C of 5f2c97e00c770d329321cff2fbd6848c into c39af132f78942f5a05423334d8ab1b6(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 02:55:38,496 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5f2c97e00c770d329321cff2fbd6848c: 2023-04-25 02:55:38,496 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c., storeName=5f2c97e00c770d329321cff2fbd6848c/C, priority=13, startTime=1682391338405; duration=0sec 2023-04-25 02:55:38,496 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 02:55:38,531 DEBUG [Thread-2900] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:55:38,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:55:38,534 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:55:38,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:55:38,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:55:38,547 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:55:38,547 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:38,547 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:55:38,547 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:55:38,548 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:38,548 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:38,549 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:38,549 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:38,549 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:38,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:38,550 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:38,550 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:55:38,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:38,550 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:38,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:55:38,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:38,550 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:55:38,552 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:55:38,552 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:55:38,552 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:55:38,552 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:38,552 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:55:38,554 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:55:38,554 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:55:38,554 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. 2023-04-25 02:55:38,554 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. started... 2023-04-25 02:55:38,554 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. as already flushing 2023-04-25 02:55:38,554 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. 2023-04-25 02:55:38,554 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:55:38,554 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:55:38,554 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:38,554 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:55:38,554 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:55:38,556 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:38,556 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:38,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:38,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:38,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:38,557 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:38,557 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:55:38,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:38,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:38,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:38,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:38,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:38,558 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:55:38,558 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4566ddfa[Count = 0] remaining members to acquire global barrier 2023-04-25 02:55:38,558 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:55:38,558 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:38,561 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:38,561 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:38,561 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:55:38,561 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:38,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:38,561 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:55:38,561 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:55:38,561 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:55:38,563 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:38,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:38,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:38,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:38,564 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:55:38,564 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:38,564 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:55:38,565 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:38,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59114 deadline: 1682391398563, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:38,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:38,566 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:38,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59116 deadline: 1682391398564, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:38,566 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:38,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59162 deadline: 1682391398564, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:38,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:38,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:38,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:38,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:38,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:38,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:38,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:55:38,569 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:55:38,569 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:55:38,569 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:55:38,570 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:55:38,570 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:55:38,570 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:55:38,571 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:38,571 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:38,571 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:38,571 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:38,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:38,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:38,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:38,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:38,572 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:38,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:38,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:38,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:38,572 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:38,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59156 deadline: 1682391398567, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:38,573 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:38,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59136 deadline: 1682391398568, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:38,573 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:38,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:38,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:38,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:38,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:38,574 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:38,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:38,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:38,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:38,577 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:38,577 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:38,577 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:38,577 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:38,577 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:38,577 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:38,577 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:38,577 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:38,577 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:38,577 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:38,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:38,577 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:38,577 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:55:38,577 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:55:38,578 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:38,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:38,578 DEBUG [Thread-2900] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:55:38,578 DEBUG [Thread-2900] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:55:38,578 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:55:38,678 DEBUG [Thread-2900] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:55:38,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:55:38,680 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:55:38,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:55:38,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:55:38,701 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:55:38,701 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:38,701 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:55:38,701 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:55:38,702 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:38,702 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:38,703 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:38,704 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:38,704 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:38,704 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:55:38,704 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:38,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:38,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:38,705 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:38,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:55:38,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:38,705 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:55:38,707 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:55:38,707 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:55:38,707 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:55:38,707 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:38,707 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:55:38,708 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:55:38,708 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:55:38,708 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. 2023-04-25 02:55:38,708 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. started... 2023-04-25 02:55:38,708 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. as already flushing 2023-04-25 02:55:38,708 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. 2023-04-25 02:55:38,708 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:55:38,708 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:55:38,708 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:38,708 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:55:38,708 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:55:38,717 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:38,717 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:38,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:38,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:38,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:38,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:38,717 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:38,718 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:55:38,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:38,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:38,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:38,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:38,719 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:55:38,719 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1b32b584[Count = 0] remaining members to acquire global barrier 2023-04-25 02:55:38,719 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:55:38,719 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:38,720 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:38,720 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:38,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:38,720 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:55:38,720 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:55:38,720 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:55:38,720 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:38,720 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:55:38,722 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:38,723 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:55:38,723 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:38,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:38,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:38,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:38,723 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:55:38,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:38,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:38,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:38,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:38,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:38,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:38,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:38,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:55:38,725 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:55:38,725 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:55:38,725 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:55:38,725 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:55:38,725 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:55:38,725 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:55:38,727 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:38,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:38,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:38,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:38,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:38,727 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:38,727 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:38,727 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:38,727 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:38,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:38,728 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:38,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:38,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:38,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:38,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:38,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:38,729 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:38,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:38,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:38,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:38,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:38,734 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:38,734 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:38,734 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:38,734 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:38,734 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:38,734 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:55:38,735 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:38,735 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:55:38,735 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:38,735 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:38,734 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:38,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:38,734 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:38,735 DEBUG [Thread-2900] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:55:38,735 DEBUG [Thread-2900] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:55:38,735 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:55:38,735 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:38,735 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:38,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:38,773 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:38,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59114 deadline: 1682391398768, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:38,774 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:38,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59116 deadline: 1682391398768, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:38,774 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:38,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59162 deadline: 1682391398768, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:38,776 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:38,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59156 deadline: 1682391398773, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:38,782 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:38,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59136 deadline: 1682391398776, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:38,835 DEBUG [Thread-2900] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:55:38,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:55:38,837 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:55:38,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:55:38,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:55:38,840 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:55:38,840 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:38,840 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:55:38,840 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:55:38,840 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:38,840 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:38,842 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:38,842 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:38,842 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:38,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:38,842 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp/A/cac2fa2cedc34beabbd5ee13b997478c as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/cac2fa2cedc34beabbd5ee13b997478c 2023-04-25 02:55:38,842 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:38,842 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:55:38,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:38,842 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:38,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:55:38,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:38,843 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:55:38,843 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:55:38,843 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:55:38,843 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:55:38,843 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:38,843 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:55:38,844 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:55:38,844 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:55:38,844 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. 2023-04-25 02:55:38,844 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. started... 2023-04-25 02:55:38,844 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. as already flushing 2023-04-25 02:55:38,844 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. 2023-04-25 02:55:38,844 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:55:38,844 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:55:38,844 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:38,844 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:55:38,844 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:55:38,846 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:38,846 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:38,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:38,846 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 5f2c97e00c770d329321cff2fbd6848c/A of 5f2c97e00c770d329321cff2fbd6848c into cac2fa2cedc34beabbd5ee13b997478c(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 02:55:38,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:38,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:38,846 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5f2c97e00c770d329321cff2fbd6848c: 2023-04-25 02:55:38,846 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c., storeName=5f2c97e00c770d329321cff2fbd6848c/A, priority=13, startTime=1682391338405; duration=0sec 2023-04-25 02:55:38,846 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 02:55:38,846 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:38,847 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:55:38,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:38,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:38,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:38,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:38,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:38,848 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:55:38,849 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6e03461c[Count = 0] remaining members to acquire global barrier 2023-04-25 02:55:38,849 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:55:38,849 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:38,850 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:38,850 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:38,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:38,850 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:55:38,850 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:55:38,850 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:55:38,850 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:38,850 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:55:38,852 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:38,852 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:55:38,852 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:38,852 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:55:38,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:38,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:38,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:38,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:38,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:38,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:38,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:38,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:38,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:38,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:38,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:55:38,855 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:55:38,855 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:55:38,855 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:55:38,855 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:55:38,855 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:55:38,855 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:55:38,857 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:38,857 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:38,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:38,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:38,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:38,857 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:38,857 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:38,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:38,857 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:38,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:38,859 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:38,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:38,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:38,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:38,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:38,859 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:38,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:38,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:38,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:38,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:38,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:38,863 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=80 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp/A/23e4830258f74dbba9a73b11c9d5f826 2023-04-25 02:55:38,864 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:38,864 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:38,864 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:38,864 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:38,864 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:38,864 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:38,864 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:55:38,864 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:55:38,864 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:38,864 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:38,864 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:38,865 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:38,865 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:38,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:38,865 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:55:38,865 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:38,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:38,866 DEBUG [Thread-2900] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:55:38,866 DEBUG [Thread-2900] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:55:38,966 DEBUG [Thread-2900] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:55:38,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:55:38,968 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:55:38,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:55:38,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:55:38,970 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:55:38,970 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:38,970 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:55:38,970 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:55:38,970 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:38,970 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:38,971 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:38,971 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:38,971 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:38,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:38,972 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:38,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:38,972 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:55:38,972 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:38,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:55:38,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:38,973 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:55:38,979 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:55:38,979 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:55:38,981 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:55:38,981 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:38,981 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:55:38,984 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:55:38,984 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:55:38,984 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. 2023-04-25 02:55:38,985 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. started... 2023-04-25 02:55:38,987 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. as already flushing 2023-04-25 02:55:38,987 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. 2023-04-25 02:55:38,987 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:55:38,987 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:55:38,987 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:38,987 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:55:38,988 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:55:38,989 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:38,989 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:38,990 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:38,990 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:55:38,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:38,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:38,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:38,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:38,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:38,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:38,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:38,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:38,995 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:55:38,995 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@352fa9f6[Count = 0] remaining members to acquire global barrier 2023-04-25 02:55:38,995 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:55:38,996 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:38,998 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:38,998 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:38,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:38,998 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:55:38,998 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:55:38,998 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:55:38,998 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:38,998 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:55:39,000 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:39,000 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:55:39,000 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:39,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:39,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:39,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:39,000 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:55:39,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:39,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:39,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:39,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:39,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:39,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:39,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:39,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:55:39,002 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:55:39,002 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:55:39,002 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:55:39,002 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:55:39,002 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:55:39,002 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:55:39,004 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:39,004 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:39,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:39,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:39,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:39,004 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:39,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:39,004 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:39,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:39,004 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:39,004 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:39,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:39,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:39,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:39,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:39,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:39,006 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:39,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:39,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:39,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:39,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:39,008 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:39,008 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:39,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:39,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:39,008 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:39,008 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:39,008 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:39,008 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:55:39,008 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:55:39,008 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:39,008 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:39,008 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:39,008 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:39,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:39,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:39,008 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:39,009 DEBUG [Thread-2900] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:55:39,009 DEBUG [Thread-2900] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:55:39,008 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:55:39,077 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:39,077 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59114 deadline: 1682391399074, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:39,079 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:39,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59116 deadline: 1682391399076, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:39,079 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:39,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59162 deadline: 1682391399076, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:39,082 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:39,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59156 deadline: 1682391399079, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:39,087 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:39,087 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59136 deadline: 1682391399085, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:39,109 DEBUG [Thread-2900] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:55:39,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:55:39,111 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:55:39,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:55:39,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:55:39,112 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:55:39,112 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:39,112 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:55:39,112 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:55:39,112 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:39,112 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:39,114 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:39,114 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:39,114 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:39,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:39,115 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:39,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:39,115 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:55:39,115 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:39,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:55:39,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:39,115 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:55:39,116 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:55:39,116 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:55:39,116 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:55:39,116 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:39,116 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:55:39,116 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:55:39,116 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:55:39,116 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. 2023-04-25 02:55:39,116 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. started... 2023-04-25 02:55:39,116 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. as already flushing 2023-04-25 02:55:39,116 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. 2023-04-25 02:55:39,116 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:55:39,116 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:55:39,116 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:39,116 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:55:39,116 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:55:39,117 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:39,117 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:39,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:39,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:39,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:39,118 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:39,118 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:55:39,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:39,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:39,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:39,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:39,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:39,119 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:55:39,119 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@46552ba4[Count = 0] remaining members to acquire global barrier 2023-04-25 02:55:39,119 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:55:39,119 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:39,120 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:39,120 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:39,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:39,120 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:55:39,120 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:55:39,120 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:55:39,120 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:39,121 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:55:39,122 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:39,122 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:55:39,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:39,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:39,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:39,122 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:39,122 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:55:39,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:39,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:39,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:39,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:39,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:39,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:39,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:39,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:55:39,124 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:55:39,124 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:55:39,124 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:55:39,124 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:55:39,124 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:55:39,124 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:55:39,126 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:39,126 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:39,126 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:39,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:39,126 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:39,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:39,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:39,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:39,126 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:39,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:39,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:39,127 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:39,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:39,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:39,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:39,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:39,128 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:39,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:39,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:39,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:39,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:39,130 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:39,130 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:39,130 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:39,130 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:39,130 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:39,130 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:55:39,131 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:55:39,130 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:39,131 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:39,131 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:39,131 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:39,131 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:39,131 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:39,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:39,131 DEBUG [Thread-2900] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:55:39,131 DEBUG [Thread-2900] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:55:39,132 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:39,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:39,132 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:55:39,190 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-25 02:55:39,231 DEBUG [Thread-2900] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:55:39,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:55:39,233 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:55:39,234 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:55:39,234 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:55:39,234 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:55:39,234 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:39,234 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:55:39,234 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:55:39,235 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:39,235 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:39,236 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:39,236 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:39,236 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:39,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:39,236 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:39,236 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:55:39,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:39,237 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:39,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:55:39,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:39,237 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:55:39,237 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:55:39,237 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:55:39,237 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:39,238 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:55:39,238 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:55:39,238 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:55:39,238 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. 2023-04-25 02:55:39,238 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. started... 2023-04-25 02:55:39,238 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. as already flushing 2023-04-25 02:55:39,238 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. 2023-04-25 02:55:39,238 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:55:39,238 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:55:39,238 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:39,238 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:55:39,238 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:55:39,239 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:39,239 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:39,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:39,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:39,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:39,240 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:39,240 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:55:39,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:39,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:39,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:39,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:39,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:39,241 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:55:39,241 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4845387f[Count = 0] remaining members to acquire global barrier 2023-04-25 02:55:39,241 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:55:39,241 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:39,243 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:39,243 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:39,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:39,243 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:55:39,243 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:55:39,243 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:55:39,243 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:39,243 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:55:39,245 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:39,245 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:55:39,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:39,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:39,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:39,245 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:39,245 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:55:39,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:39,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:39,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:39,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:39,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:39,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:39,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:39,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:55:39,248 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:55:39,248 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:55:39,248 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:55:39,248 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:55:39,248 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:55:39,248 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:55:39,250 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:39,250 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:39,250 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:39,250 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:39,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:39,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:39,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:39,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:39,250 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:39,250 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:39,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:39,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:39,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:39,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:39,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:39,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:39,252 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:39,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:39,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:39,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:39,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:39,254 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:39,254 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:39,254 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:39,254 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:39,254 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:39,254 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:39,254 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:39,254 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:39,254 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:55:39,254 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:55:39,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:39,254 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:39,254 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:39,255 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:39,255 DEBUG [Thread-2900] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:55:39,255 DEBUG [Thread-2900] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:55:39,255 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:39,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:39,287 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=80 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp/B/2410eb188561495fa42936ef537ce3f9 2023-04-25 02:55:39,302 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=80 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp/C/a537c55d4a7b4b2990fa3405e4faa39b 2023-04-25 02:55:39,305 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp/A/23e4830258f74dbba9a73b11c9d5f826 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/23e4830258f74dbba9a73b11c9d5f826 2023-04-25 02:55:39,308 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/23e4830258f74dbba9a73b11c9d5f826, entries=200, sequenceid=80, filesize=13.9 K 2023-04-25 02:55:39,309 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp/B/2410eb188561495fa42936ef537ce3f9 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/2410eb188561495fa42936ef537ce3f9 2023-04-25 02:55:39,313 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/2410eb188561495fa42936ef537ce3f9, entries=150, sequenceid=80, filesize=11.6 K 2023-04-25 02:55:39,316 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp/C/a537c55d4a7b4b2990fa3405e4faa39b as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/a537c55d4a7b4b2990fa3405e4faa39b 2023-04-25 02:55:39,322 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/a537c55d4a7b4b2990fa3405e4faa39b, entries=150, sequenceid=80, filesize=11.6 K 2023-04-25 02:55:39,322 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 5f2c97e00c770d329321cff2fbd6848c in 881ms, sequenceid=80, compaction requested=false 2023-04-25 02:55:39,322 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 5f2c97e00c770d329321cff2fbd6848c: 2023-04-25 02:55:39,355 DEBUG [Thread-2900] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:55:39,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:55:39,358 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:55:39,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:55:39,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:55:39,360 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:55:39,360 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:39,360 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:55:39,360 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:55:39,361 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:39,361 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:39,366 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:39,366 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:39,366 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:39,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:39,366 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:39,366 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:55:39,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:39,367 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:39,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:55:39,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:39,367 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:55:39,368 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:55:39,368 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:55:39,368 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:55:39,368 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:39,368 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:55:39,368 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:55:39,368 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:55:39,368 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. 2023-04-25 02:55:39,368 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. started... 2023-04-25 02:55:39,368 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5f2c97e00c770d329321cff2fbd6848c 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-25 02:55:39,368 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5f2c97e00c770d329321cff2fbd6848c, store=A 2023-04-25 02:55:39,369 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:55:39,369 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5f2c97e00c770d329321cff2fbd6848c, store=B 2023-04-25 02:55:39,369 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:55:39,369 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5f2c97e00c770d329321cff2fbd6848c, store=C 2023-04-25 02:55:39,369 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:55:39,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(9158): Flush requested on 5f2c97e00c770d329321cff2fbd6848c 2023-04-25 02:55:39,584 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. as already flushing 2023-04-25 02:55:39,605 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:39,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59156 deadline: 1682391399597, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:39,610 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:39,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59114 deadline: 1682391399603, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:39,610 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:39,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59136 deadline: 1682391399604, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:39,610 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:39,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59116 deadline: 1682391399604, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:39,611 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:39,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59162 deadline: 1682391399605, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:39,711 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:39,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59156 deadline: 1682391399706, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:39,715 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:39,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59114 deadline: 1682391399711, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:39,716 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:39,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59136 deadline: 1682391399711, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:39,716 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:39,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59116 deadline: 1682391399711, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:39,716 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:39,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59162 deadline: 1682391399712, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:39,784 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=95 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp/A/329c77bde63b4a60bcaba2a4365c7d99 2023-04-25 02:55:39,918 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:39,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59156 deadline: 1682391399913, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:39,922 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:39,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59114 deadline: 1682391399916, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:39,923 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:39,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59116 deadline: 1682391399917, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:39,923 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:39,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59136 deadline: 1682391399917, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:39,923 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:39,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59162 deadline: 1682391399918, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:40,195 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=95 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp/B/1e555b288f224498a7ca167aaa30d2ad 2023-04-25 02:55:40,211 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=95 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp/C/05f9721c3c724ab0a2ac4f8dcd81f1a4 2023-04-25 02:55:40,217 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp/A/329c77bde63b4a60bcaba2a4365c7d99 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/329c77bde63b4a60bcaba2a4365c7d99 2023-04-25 02:55:40,225 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:40,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59156 deadline: 1682391400221, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:40,225 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/329c77bde63b4a60bcaba2a4365c7d99, entries=150, sequenceid=95, filesize=11.6 K 2023-04-25 02:55:40,226 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp/B/1e555b288f224498a7ca167aaa30d2ad as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/1e555b288f224498a7ca167aaa30d2ad 2023-04-25 02:55:40,237 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/1e555b288f224498a7ca167aaa30d2ad, entries=150, sequenceid=95, filesize=11.6 K 2023-04-25 02:55:40,240 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp/C/05f9721c3c724ab0a2ac4f8dcd81f1a4 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/05f9721c3c724ab0a2ac4f8dcd81f1a4 2023-04-25 02:55:40,240 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:40,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59136 deadline: 1682391400224, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:40,240 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:40,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59162 deadline: 1682391400224, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:40,244 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:40,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59114 deadline: 1682391400225, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:40,245 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:40,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59116 deadline: 1682391400226, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:40,247 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/05f9721c3c724ab0a2ac4f8dcd81f1a4, entries=150, sequenceid=95, filesize=11.6 K 2023-04-25 02:55:40,248 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 5f2c97e00c770d329321cff2fbd6848c in 880ms, sequenceid=95, compaction requested=true 2023-04-25 02:55:40,249 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5f2c97e00c770d329321cff2fbd6848c: 2023-04-25 02:55:40,249 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. 2023-04-25 02:55:40,249 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:55:40,249 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:55:40,249 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:40,251 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:55:40,251 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:55:40,252 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:40,252 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:40,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:40,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:40,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:40,253 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:40,253 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:55:40,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:40,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:40,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:40,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:40,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:40,254 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:55:40,254 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3c02a4cd[Count = 0] remaining members to acquire global barrier 2023-04-25 02:55:40,254 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:55:40,254 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:40,255 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:40,255 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:40,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:40,255 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:55:40,256 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:55:40,256 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:55:40,256 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:40,256 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:55:40,258 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:55:40,258 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:40,258 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:40,258 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:55:40,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:40,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:40,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:40,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:40,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:40,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:40,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:40,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:40,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:40,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:40,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:55:40,262 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:55:40,262 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:55:40,262 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:55:40,262 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:55:40,262 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:55:40,262 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:55:40,263 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:40,263 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:40,263 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:40,263 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:40,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:40,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:40,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:40,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:40,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:40,264 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:40,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:40,265 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:40,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:40,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:40,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:40,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:40,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:40,267 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:40,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:40,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:40,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:40,269 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:40,269 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:40,269 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:40,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:40,269 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:40,269 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:40,269 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:40,269 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:40,269 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:55:40,270 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:55:40,269 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:40,270 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:40,269 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:40,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:40,270 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:40,270 DEBUG [Thread-2900] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:55:40,270 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:40,270 DEBUG [Thread-2900] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:55:40,370 DEBUG [Thread-2900] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:55:40,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:55:40,372 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:55:40,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:55:40,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:55:40,373 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:55:40,373 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:40,374 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:55:40,374 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:55:40,374 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:40,374 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:40,375 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:40,375 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:40,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:40,375 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:40,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:40,376 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:40,376 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:55:40,376 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:40,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:55:40,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:40,377 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:55:40,384 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:55:40,384 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:55:40,384 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:55:40,384 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:40,384 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:55:40,387 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:55:40,387 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:55:40,387 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. 2023-04-25 02:55:40,387 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. started... 2023-04-25 02:55:40,387 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5f2c97e00c770d329321cff2fbd6848c 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-25 02:55:40,388 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5f2c97e00c770d329321cff2fbd6848c, store=A 2023-04-25 02:55:40,388 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:55:40,388 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5f2c97e00c770d329321cff2fbd6848c, store=B 2023-04-25 02:55:40,388 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:55:40,388 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5f2c97e00c770d329321cff2fbd6848c, store=C 2023-04-25 02:55:40,388 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:55:40,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(9158): Flush requested on 5f2c97e00c770d329321cff2fbd6848c 2023-04-25 02:55:40,734 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. as already flushing 2023-04-25 02:55:40,755 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:40,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59136 deadline: 1682391400750, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:40,758 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:40,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59162 deadline: 1682391400751, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:40,758 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:40,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59114 deadline: 1682391400752, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:40,758 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:40,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59156 deadline: 1682391400753, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:40,759 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:40,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59116 deadline: 1682391400755, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:40,819 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=115 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp/A/1713671593824abd8ab8ddb68b32e1b7 2023-04-25 02:55:40,835 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=115 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp/B/c38c46225b7749fb978d49bdf27c4e26 2023-04-25 02:55:40,851 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=115 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp/C/14a6f6a7152b44e48dc44f684cf6f6d6 2023-04-25 02:55:40,861 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp/A/1713671593824abd8ab8ddb68b32e1b7 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/1713671593824abd8ab8ddb68b32e1b7 2023-04-25 02:55:40,863 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:40,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59136 deadline: 1682391400856, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:40,863 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:40,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59114 deadline: 1682391400859, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:40,864 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:40,864 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59156 deadline: 1682391400859, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:40,864 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:40,864 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59162 deadline: 1682391400859, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:40,864 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:40,864 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59116 deadline: 1682391400860, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:40,865 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/1713671593824abd8ab8ddb68b32e1b7, entries=150, sequenceid=115, filesize=11.6 K 2023-04-25 02:55:40,866 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp/B/c38c46225b7749fb978d49bdf27c4e26 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/c38c46225b7749fb978d49bdf27c4e26 2023-04-25 02:55:40,877 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/c38c46225b7749fb978d49bdf27c4e26, entries=150, sequenceid=115, filesize=11.6 K 2023-04-25 02:55:40,878 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp/C/14a6f6a7152b44e48dc44f684cf6f6d6 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/14a6f6a7152b44e48dc44f684cf6f6d6 2023-04-25 02:55:40,898 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/14a6f6a7152b44e48dc44f684cf6f6d6, entries=150, sequenceid=115, filesize=11.6 K 2023-04-25 02:55:40,899 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 5f2c97e00c770d329321cff2fbd6848c in 512ms, sequenceid=115, compaction requested=true 2023-04-25 02:55:40,899 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5f2c97e00c770d329321cff2fbd6848c: 2023-04-25 02:55:40,900 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. 2023-04-25 02:55:40,900 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:55:40,900 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:55:40,900 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:40,900 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:55:40,900 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:55:40,904 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:40,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:40,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:40,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:40,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:40,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:40,920 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:40,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:40,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:40,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:40,921 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:40,921 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:55:40,922 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:55:40,922 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@742adaa2[Count = 0] remaining members to acquire global barrier 2023-04-25 02:55:40,922 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:55:40,922 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:40,924 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:40,924 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:40,924 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:55:40,924 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:40,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:40,924 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:55:40,924 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:55:40,924 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:55:40,925 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:40,926 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:55:40,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:40,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:40,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:40,926 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:40,926 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:55:40,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:40,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:40,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:40,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:40,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:40,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:40,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:40,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:55:40,928 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:55:40,928 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:55:40,928 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:55:40,928 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:55:40,928 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:55:40,929 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:55:40,930 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:40,930 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:40,930 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:40,930 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:40,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:40,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:40,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:40,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:40,931 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:40,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:40,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:40,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:40,931 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:40,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:40,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:40,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:40,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:40,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:40,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:40,933 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:40,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:40,936 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:40,936 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:40,936 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:40,936 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:40,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:40,936 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:40,936 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:40,936 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:40,936 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:40,936 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:40,936 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:40,937 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:40,937 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:55:40,937 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:55:40,938 DEBUG [Thread-2900] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:55:40,938 DEBUG [Thread-2900] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:55:40,938 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:40,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:41,038 DEBUG [Thread-2900] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:55:41,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:55:41,041 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:55:41,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:55:41,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:55:41,042 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:55:41,043 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:41,043 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:55:41,043 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:55:41,043 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:41,043 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:41,044 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:41,044 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:41,044 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:41,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:41,045 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:41,045 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:55:41,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:41,045 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:41,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:55:41,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:41,046 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:55:41,046 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:55:41,046 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:55:41,046 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:55:41,046 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:41,046 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:55:41,047 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:55:41,047 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:55:41,047 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. 2023-04-25 02:55:41,047 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. started... 2023-04-25 02:55:41,047 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5f2c97e00c770d329321cff2fbd6848c 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-25 02:55:41,047 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5f2c97e00c770d329321cff2fbd6848c, store=A 2023-04-25 02:55:41,047 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:55:41,047 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5f2c97e00c770d329321cff2fbd6848c, store=B 2023-04-25 02:55:41,048 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:55:41,048 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5f2c97e00c770d329321cff2fbd6848c, store=C 2023-04-25 02:55:41,048 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:55:41,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(9158): Flush requested on 5f2c97e00c770d329321cff2fbd6848c 2023-04-25 02:55:41,075 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. as already flushing 2023-04-25 02:55:41,141 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:41,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59116 deadline: 1682391401139, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:41,146 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:41,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59162 deadline: 1682391401139, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:41,146 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:41,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59156 deadline: 1682391401140, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:41,146 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:41,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59114 deadline: 1682391401141, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:41,146 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:41,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59136 deadline: 1682391401142, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:41,247 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:41,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59116 deadline: 1682391401243, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:41,251 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:41,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59162 deadline: 1682391401247, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:41,252 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:41,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59114 deadline: 1682391401247, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:41,252 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:41,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59156 deadline: 1682391401247, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:41,252 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:41,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59136 deadline: 1682391401247, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:41,454 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:41,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59116 deadline: 1682391401448, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:41,457 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:41,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59162 deadline: 1682391401452, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:41,457 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:41,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59156 deadline: 1682391401453, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:41,457 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:41,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59114 deadline: 1682391401453, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:41,461 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:41,461 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59136 deadline: 1682391401455, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:41,471 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=127 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp/A/25bb38aa1ffa4896a8c2765283fe8746 2023-04-25 02:55:41,683 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] regionserver.HStore(2712): Moving the files [hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/978bb4c158c9443b964b311f5b9fdcb9, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/8c4835a8c95c4d9eacfb721cd6590da2, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/9752f6e742dd42a695156126d7f9adb9] to archive 2023-04-25 02:55:41,684 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(360): Archiving compacted files. 2023-04-25 02:55:41,686 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/978bb4c158c9443b964b311f5b9fdcb9 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/978bb4c158c9443b964b311f5b9fdcb9 2023-04-25 02:55:41,687 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/8c4835a8c95c4d9eacfb721cd6590da2 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/8c4835a8c95c4d9eacfb721cd6590da2 2023-04-25 02:55:41,688 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/9752f6e742dd42a695156126d7f9adb9 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/9752f6e742dd42a695156126d7f9adb9 2023-04-25 02:55:41,694 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] regionserver.HStore(2712): Moving the files [hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/2558ca7b08414e49a9823e8510998a7e, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/69850179080a482fb3fe733a789862c5, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/40fbc1e0fc064f6283aa283f185ceb0d] to archive 2023-04-25 02:55:41,695 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(360): Archiving compacted files. 2023-04-25 02:55:41,696 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/2558ca7b08414e49a9823e8510998a7e to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/2558ca7b08414e49a9823e8510998a7e 2023-04-25 02:55:41,697 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/69850179080a482fb3fe733a789862c5 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/69850179080a482fb3fe733a789862c5 2023-04-25 02:55:41,698 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/40fbc1e0fc064f6283aa283f185ceb0d to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/40fbc1e0fc064f6283aa283f185ceb0d 2023-04-25 02:55:41,704 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] regionserver.HStore(2712): Moving the files [hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/90075ab23b324726b3123f4930955837, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/1846ea21c79a487f821d3bbecf00b000, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/fb6b837b3def4a1c88bbb59f078426d6] to archive 2023-04-25 02:55:41,705 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(360): Archiving compacted files. 2023-04-25 02:55:41,706 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/90075ab23b324726b3123f4930955837 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/90075ab23b324726b3123f4930955837 2023-04-25 02:55:41,707 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/1846ea21c79a487f821d3bbecf00b000 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/1846ea21c79a487f821d3bbecf00b000 2023-04-25 02:55:41,708 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/fb6b837b3def4a1c88bbb59f078426d6 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/fb6b837b3def4a1c88bbb59f078426d6 2023-04-25 02:55:41,764 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:41,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59162 deadline: 1682391401758, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:41,767 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:41,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59156 deadline: 1682391401759, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:41,767 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:41,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59114 deadline: 1682391401759, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:41,767 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:41,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59116 deadline: 1682391401760, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:41,775 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:41,775 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59136 deadline: 1682391401765, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:41,881 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=127 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp/B/041e58301c0042749998a15ae582afbb 2023-04-25 02:55:42,272 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:42,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59156 deadline: 1682391402268, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:42,272 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:42,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59162 deadline: 1682391402269, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:42,272 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:42,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59114 deadline: 1682391402270, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:42,277 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:42,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59116 deadline: 1682391402272, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:42,283 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:42,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59136 deadline: 1682391402280, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:42,301 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=127 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp/C/bc783b0c9dc54965a05b5ae9c1b9a730 2023-04-25 02:55:42,314 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp/A/25bb38aa1ffa4896a8c2765283fe8746 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/25bb38aa1ffa4896a8c2765283fe8746 2023-04-25 02:55:42,324 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/25bb38aa1ffa4896a8c2765283fe8746, entries=150, sequenceid=127, filesize=11.6 K 2023-04-25 02:55:42,325 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp/B/041e58301c0042749998a15ae582afbb as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/041e58301c0042749998a15ae582afbb 2023-04-25 02:55:42,329 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/041e58301c0042749998a15ae582afbb, entries=150, sequenceid=127, filesize=11.6 K 2023-04-25 02:55:42,329 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp/C/bc783b0c9dc54965a05b5ae9c1b9a730 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/bc783b0c9dc54965a05b5ae9c1b9a730 2023-04-25 02:55:42,332 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/bc783b0c9dc54965a05b5ae9c1b9a730, entries=150, sequenceid=127, filesize=11.6 K 2023-04-25 02:55:42,333 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 5f2c97e00c770d329321cff2fbd6848c in 1286ms, sequenceid=127, compaction requested=true 2023-04-25 02:55:42,333 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5f2c97e00c770d329321cff2fbd6848c: 2023-04-25 02:55:42,333 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. 2023-04-25 02:55:42,333 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:55:42,333 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:55:42,333 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:42,333 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:55:42,333 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:55:42,335 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:42,335 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:42,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:42,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:42,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:42,335 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:42,335 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:55:42,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:42,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:42,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:42,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:42,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:42,336 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:55:42,336 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1b4dd788[Count = 0] remaining members to acquire global barrier 2023-04-25 02:55:42,336 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:55:42,336 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:42,337 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:42,337 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:42,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:42,337 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:55:42,337 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:55:42,337 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:55:42,338 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:42,338 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:55:42,339 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:55:42,339 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:42,339 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:55:42,339 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:42,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:42,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:42,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:42,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:42,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:42,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:42,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:42,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:42,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:42,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:42,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:55:42,341 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:55:42,341 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:55:42,341 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:55:42,341 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:55:42,341 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:55:42,341 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:55:42,343 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:42,344 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:42,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:42,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:42,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:42,344 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:42,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:42,344 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:42,344 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:42,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:42,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:42,344 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:42,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:42,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:42,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:42,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:42,345 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:42,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:42,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:42,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:42,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:42,347 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:42,347 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:42,347 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:42,347 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:42,347 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:55:42,347 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:55:42,347 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:42,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:42,347 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:42,347 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:42,348 DEBUG [Thread-2900] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:55:42,347 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:55:42,347 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:42,348 DEBUG [Thread-2900] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:55:42,348 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:42,348 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:42,348 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:42,348 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:42,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:42,448 DEBUG [Thread-2900] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:55:42,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:55:42,450 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:55:42,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:55:42,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:55:42,451 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:55:42,451 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:42,451 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:55:42,451 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:55:42,452 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:42,452 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:42,454 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:42,454 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:42,454 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:42,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:42,454 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:42,454 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:55:42,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:42,455 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:42,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:55:42,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:42,455 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:55:42,455 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:55:42,455 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:55:42,456 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:55:42,456 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:42,456 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:55:42,456 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:55:42,456 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:55:42,456 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. 2023-04-25 02:55:42,456 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. started... 2023-04-25 02:55:42,456 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5f2c97e00c770d329321cff2fbd6848c 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-25 02:55:42,456 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5f2c97e00c770d329321cff2fbd6848c, store=A 2023-04-25 02:55:42,456 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:55:42,456 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5f2c97e00c770d329321cff2fbd6848c, store=B 2023-04-25 02:55:42,457 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:55:42,457 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5f2c97e00c770d329321cff2fbd6848c, store=C 2023-04-25 02:55:42,457 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:55:42,463 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=147 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp/A/4b83b1d55379429ba95b44494cb20aea 2023-04-25 02:55:42,879 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=147 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp/B/6439852b16434808a0763062e981e154 2023-04-25 02:55:43,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(9158): Flush requested on 5f2c97e00c770d329321cff2fbd6848c 2023-04-25 02:55:43,278 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. as already flushing 2023-04-25 02:55:43,290 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=147 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp/C/67b65249b4b04c09bdb0a8f5619d5d6e 2023-04-25 02:55:43,294 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp/A/4b83b1d55379429ba95b44494cb20aea as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/4b83b1d55379429ba95b44494cb20aea 2023-04-25 02:55:43,294 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:43,295 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59136 deadline: 1682391403286, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:43,295 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:43,295 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59156 deadline: 1682391403288, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:43,297 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:43,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59116 deadline: 1682391403289, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:43,297 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:43,297 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/4b83b1d55379429ba95b44494cb20aea, entries=150, sequenceid=147, filesize=11.8 K 2023-04-25 02:55:43,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59114 deadline: 1682391403289, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:43,298 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp/B/6439852b16434808a0763062e981e154 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/6439852b16434808a0763062e981e154 2023-04-25 02:55:43,301 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:43,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59162 deadline: 1682391403295, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:43,301 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/6439852b16434808a0763062e981e154, entries=150, sequenceid=147, filesize=11.8 K 2023-04-25 02:55:43,302 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp/C/67b65249b4b04c09bdb0a8f5619d5d6e as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/67b65249b4b04c09bdb0a8f5619d5d6e 2023-04-25 02:55:43,305 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/67b65249b4b04c09bdb0a8f5619d5d6e, entries=150, sequenceid=147, filesize=11.8 K 2023-04-25 02:55:43,305 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 5f2c97e00c770d329321cff2fbd6848c in 849ms, sequenceid=147, compaction requested=true 2023-04-25 02:55:43,305 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5f2c97e00c770d329321cff2fbd6848c: 2023-04-25 02:55:43,306 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. 2023-04-25 02:55:43,306 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:55:43,306 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:55:43,306 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:43,306 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:55:43,306 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:55:43,314 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:43,314 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:43,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:43,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:43,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:43,314 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:43,314 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:55:43,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:43,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:43,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:43,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:43,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:43,315 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:55:43,316 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7aaa925e[Count = 0] remaining members to acquire global barrier 2023-04-25 02:55:43,316 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:55:43,316 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:43,317 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:43,317 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:43,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:43,317 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:55:43,317 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:55:43,317 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:55:43,317 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:43,317 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:55:43,318 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:55:43,318 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:43,318 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:55:43,318 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:43,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:43,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:43,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:43,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:43,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:43,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:43,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:43,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:43,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:43,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:43,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:55:43,320 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:55:43,320 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:55:43,320 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:55:43,320 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:55:43,320 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:55:43,320 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:55:43,322 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:43,322 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:43,322 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:43,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:43,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:43,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:43,322 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:43,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:43,322 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:43,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:43,322 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:43,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:43,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:43,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:43,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:43,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:43,323 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:43,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:43,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:43,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:43,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:43,328 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:43,328 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:43,328 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:43,328 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:43,328 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:43,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:43,328 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:43,328 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:43,329 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:55:43,329 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:55:43,328 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:43,329 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:43,329 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:43,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:43,329 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:55:43,329 DEBUG [Thread-2900] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:55:43,329 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:43,329 DEBUG [Thread-2900] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:55:43,329 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:43,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(9158): Flush requested on 5f2c97e00c770d329321cff2fbd6848c 2023-04-25 02:55:43,400 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 5f2c97e00c770d329321cff2fbd6848c 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-25 02:55:43,400 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5f2c97e00c770d329321cff2fbd6848c, store=A 2023-04-25 02:55:43,400 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:55:43,400 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5f2c97e00c770d329321cff2fbd6848c, store=B 2023-04-25 02:55:43,400 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:55:43,400 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5f2c97e00c770d329321cff2fbd6848c, store=C 2023-04-25 02:55:43,400 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:55:43,429 DEBUG [Thread-2900] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:55:43,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:55:43,431 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:55:43,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:55:43,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:55:43,432 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:55:43,432 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:43,432 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:55:43,432 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:55:43,433 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:43,433 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:43,434 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:43,434 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:43,434 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:43,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:43,434 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:43,434 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:55:43,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:43,435 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:43,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:55:43,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:43,435 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:55:43,435 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:55:43,435 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:55:43,435 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:55:43,435 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:43,435 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:55:43,444 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:55:43,444 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:55:43,444 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. 2023-04-25 02:55:43,444 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. started... 2023-04-25 02:55:43,444 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. as already flushing 2023-04-25 02:55:43,444 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. 2023-04-25 02:55:43,444 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:55:43,444 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:55:43,444 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:43,444 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:55:43,444 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:55:43,446 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:43,446 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:43,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:43,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:43,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:43,446 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:43,446 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:55:43,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:43,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:43,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:43,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:43,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:43,447 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:55:43,447 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@53a7750a[Count = 0] remaining members to acquire global barrier 2023-04-25 02:55:43,447 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:55:43,447 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:43,448 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:43,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59116 deadline: 1682391403437, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:43,448 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:43,448 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:43,448 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:43,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:43,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59114 deadline: 1682391403438, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:43,448 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:43,448 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:55:43,448 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:55:43,449 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:55:43,448 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:43,449 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:55:43,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59156 deadline: 1682391403439, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:43,451 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:55:43,451 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:43,451 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:43,451 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:55:43,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:43,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:43,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:43,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:43,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:43,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:43,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:43,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:43,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:43,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:43,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:55:43,453 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:55:43,453 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:55:43,453 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:55:43,453 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:55:43,453 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:55:43,454 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:55:43,455 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:43,455 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:43,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:43,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:43,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:43,455 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:43,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:43,455 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:43,455 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:43,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:43,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:43,455 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:43,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:43,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:43,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:43,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:43,457 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:43,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:43,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59162 deadline: 1682391403440, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:43,457 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:43,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:43,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:43,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:43,460 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:43,460 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:43,460 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:43,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:43,460 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:43,461 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:55:43,461 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:55:43,461 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:43,461 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:43,461 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:43,461 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:43,461 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:43,461 DEBUG [Thread-2900] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:55:43,461 DEBUG [Thread-2900] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:55:43,461 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:43,461 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:43,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:43,461 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:43,555 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:43,555 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59116 deadline: 1682391403549, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:43,556 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:43,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59114 deadline: 1682391403549, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:43,556 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:43,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59156 deadline: 1682391403550, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:43,561 DEBUG [Thread-2900] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:55:43,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:55:43,566 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:55:43,566 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:43,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59162 deadline: 1682391403559, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:43,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:55:43,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:55:43,574 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:55:43,574 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:43,574 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:55:43,574 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:55:43,574 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:43,574 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:43,575 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:43,575 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:43,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:43,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:43,576 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:43,576 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:43,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:55:43,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:43,577 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:55:43,577 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:43,577 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:55:43,584 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:55:43,584 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:55:43,584 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:55:43,584 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:43,584 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:55:43,588 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:55:43,588 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:55:43,588 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. 2023-04-25 02:55:43,588 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. started... 2023-04-25 02:55:43,588 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. as already flushing 2023-04-25 02:55:43,588 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. 2023-04-25 02:55:43,588 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:55:43,588 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:55:43,588 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:43,588 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:55:43,588 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:55:43,590 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:43,590 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:43,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:43,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:43,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:43,591 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:43,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:43,591 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:55:43,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:43,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:43,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:43,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:43,592 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:55:43,592 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@717a4a16[Count = 0] remaining members to acquire global barrier 2023-04-25 02:55:43,592 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:55:43,592 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:43,593 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:43,594 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:43,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:43,594 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:55:43,594 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:55:43,594 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:55:43,596 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:55:43,596 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:43,596 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:55:43,596 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:43,597 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:55:43,597 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:55:43,597 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:55:43,597 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:55:43,597 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:55:43,597 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:55:43,597 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:55:43,599 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:43,599 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:43,599 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:43,599 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:43,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:43,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:43,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:43,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:43,600 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:43,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:43,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:43,600 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:43,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:43,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:43,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:43,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:43,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:43,602 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:43,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:43,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:43,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:43,605 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:43,605 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:43,605 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:43,605 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:43,605 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:43,605 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:43,605 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:43,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:43,605 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:43,605 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:55:43,605 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:55:43,605 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:43,605 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:43,605 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:43,605 DEBUG [Thread-2900] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:55:43,605 DEBUG [Thread-2900] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:55:43,605 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:43,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:43,605 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:55:43,705 DEBUG [Thread-2900] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:55:43,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:55:43,708 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:55:43,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:55:43,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:55:43,709 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:55:43,709 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:43,709 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:55:43,709 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:55:43,710 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:43,710 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:43,711 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:43,711 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:43,711 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:43,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:43,711 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:43,711 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:55:43,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:43,711 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:43,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:55:43,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:43,712 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:55:43,712 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:55:43,712 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:55:43,712 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:55:43,712 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:43,712 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:55:43,712 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:55:43,712 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:55:43,712 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. 2023-04-25 02:55:43,713 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. started... 2023-04-25 02:55:43,713 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. as already flushing 2023-04-25 02:55:43,713 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. 2023-04-25 02:55:43,713 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:55:43,713 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:55:43,713 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:43,713 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:55:43,713 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:55:43,714 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:43,714 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:43,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:43,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:43,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:43,714 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:43,714 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:55:43,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:43,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:43,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:43,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:43,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:43,715 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:55:43,716 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4fe2226[Count = 0] remaining members to acquire global barrier 2023-04-25 02:55:43,716 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:55:43,716 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:43,717 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:43,717 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:43,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:43,717 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:55:43,717 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:43,717 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:55:43,717 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:55:43,717 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:55:43,722 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:43,722 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:55:43,722 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:43,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:43,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:43,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:43,722 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:55:43,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:43,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:43,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:43,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:43,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:43,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:43,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:43,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:55:43,724 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:55:43,724 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:55:43,724 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:55:43,724 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:55:43,724 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:55:43,724 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:55:43,726 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:43,726 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:43,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:43,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:43,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:43,726 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:43,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:43,726 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:43,726 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:43,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:43,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:43,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:43,727 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:43,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:43,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:43,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:43,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:43,728 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:43,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:43,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:43,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:43,731 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:43,731 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:43,731 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:43,731 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:43,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:43,731 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:55:43,731 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:55:43,731 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:43,731 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:43,731 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:43,731 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:43,731 DEBUG [Thread-2900] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:55:43,731 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:43,731 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:43,731 DEBUG [Thread-2900] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:55:43,731 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:43,731 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:43,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:43,764 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:43,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59116 deadline: 1682391403756, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:43,764 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:43,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59114 deadline: 1682391403758, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:43,764 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:43,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59156 deadline: 1682391403758, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:43,781 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:43,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59162 deadline: 1682391403768, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:43,807 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=160 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp/A/fbe6297e3cf8405cad592ff83411a75d 2023-04-25 02:55:43,832 DEBUG [Thread-2900] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:55:43,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:55:43,834 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:55:43,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:55:43,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:55:43,868 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:55:43,868 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:43,868 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:55:43,868 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:55:43,869 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:43,869 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:43,870 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:43,870 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:43,870 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:43,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:43,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:43,871 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:43,871 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:55:43,871 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:43,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:55:43,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:43,872 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:55:43,872 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:55:43,872 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:55:43,872 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:55:43,872 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:43,872 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:55:43,880 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:55:43,880 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:55:43,880 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. 2023-04-25 02:55:43,880 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. started... 2023-04-25 02:55:43,880 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. as already flushing 2023-04-25 02:55:43,880 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. 2023-04-25 02:55:43,880 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:55:43,880 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:55:43,880 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:43,880 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:55:43,880 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:55:43,882 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:43,883 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:43,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:43,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:43,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:43,883 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:43,883 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:55:43,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:43,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:43,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:43,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:43,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:43,884 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:55:43,884 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2ae87885[Count = 0] remaining members to acquire global barrier 2023-04-25 02:55:43,884 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:55:43,884 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:43,885 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:43,885 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:43,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:43,885 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:55:43,886 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:55:43,886 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:55:43,886 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:43,886 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:55:43,887 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:55:43,888 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:43,888 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:55:43,888 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:43,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:43,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:43,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:43,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:43,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:43,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:43,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:43,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:43,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:43,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:43,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:55:43,890 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:55:43,890 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:55:43,890 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:55:43,890 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:55:43,890 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:55:43,890 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:55:43,891 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:43,891 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:43,891 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:43,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:43,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:43,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:43,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:43,892 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:43,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:43,891 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:43,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:43,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:43,893 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:43,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:43,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:43,893 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:43,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:43,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:43,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:43,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:43,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:43,897 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:43,897 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:43,897 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:43,897 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:43,897 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:43,897 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:43,897 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:43,898 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:43,898 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:43,898 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:55:43,898 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:55:43,898 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:43,898 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:43,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:43,898 DEBUG [Thread-2900] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:55:43,898 DEBUG [Thread-2900] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:55:43,898 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:55:43,898 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:43,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:43,998 DEBUG [Thread-2900] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:55:43,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:55:44,001 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:55:44,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:55:44,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:55:44,002 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:55:44,002 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:44,002 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:55:44,002 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:55:44,002 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:44,002 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:44,004 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:44,004 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:44,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:44,004 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:44,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:44,004 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:44,004 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:55:44,004 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:44,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:55:44,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:44,005 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:55:44,005 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:55:44,005 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:55:44,005 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:55:44,005 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:44,005 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:55:44,006 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:55:44,006 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:55:44,006 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. 2023-04-25 02:55:44,006 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. started... 2023-04-25 02:55:44,007 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. as already flushing 2023-04-25 02:55:44,007 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. 2023-04-25 02:55:44,007 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:55:44,007 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:55:44,007 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:44,007 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:55:44,007 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:55:44,008 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:44,008 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:44,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:44,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:44,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:44,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:44,009 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:44,009 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:55:44,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:44,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:44,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:44,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:44,011 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:55:44,011 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@428f57f7[Count = 0] remaining members to acquire global barrier 2023-04-25 02:55:44,011 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:55:44,011 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:44,012 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:44,012 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:44,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:44,012 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:55:44,012 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:55:44,012 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:55:44,012 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:44,012 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:55:44,014 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:55:44,014 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:44,014 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:44,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:44,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:44,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:44,014 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:55:44,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:44,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:44,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:44,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:44,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:44,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:44,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:44,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:55:44,016 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:55:44,016 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:55:44,016 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:55:44,017 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:55:44,017 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:55:44,017 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:55:44,018 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:44,018 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:44,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:44,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:44,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:44,018 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:44,018 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:44,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:44,018 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:44,018 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:44,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:44,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:44,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:44,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:44,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:44,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:44,021 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:44,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:44,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:44,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:44,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:44,023 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:44,023 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:44,023 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:44,023 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:44,023 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:55:44,023 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:55:44,023 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:44,023 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:55:44,023 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:44,023 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:44,023 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:44,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:44,023 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:44,023 DEBUG [Thread-2900] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:55:44,023 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:44,023 DEBUG [Thread-2900] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:55:44,023 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:44,023 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:44,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:44,066 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:44,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59114 deadline: 1682391404066, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:44,074 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:44,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59116 deadline: 1682391404067, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:44,074 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:44,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59156 deadline: 1682391404067, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:44,085 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:44,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59162 deadline: 1682391404083, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:44,123 DEBUG [Thread-2900] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:55:44,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:55:44,126 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:55:44,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:55:44,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:55:44,127 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:55:44,127 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:44,127 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:55:44,127 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:55:44,127 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:44,127 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:44,129 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:44,129 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:44,129 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:44,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:44,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:44,129 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:44,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:55:44,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:44,130 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:55:44,130 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:55:44,130 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:55:44,130 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:44,130 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:55:44,130 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:55:44,130 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:44,130 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:55:44,131 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:55:44,131 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:55:44,131 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. 2023-04-25 02:55:44,131 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. started... 2023-04-25 02:55:44,131 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. as already flushing 2023-04-25 02:55:44,131 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. 2023-04-25 02:55:44,131 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:55:44,131 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:55:44,131 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:44,131 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:55:44,131 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:55:44,134 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:44,134 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:44,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:44,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:44,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:44,134 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:44,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:44,134 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:55:44,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:44,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:44,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:44,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:44,136 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:55:44,136 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@61068f4a[Count = 0] remaining members to acquire global barrier 2023-04-25 02:55:44,136 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:55:44,136 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:44,137 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:44,137 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:44,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:44,137 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:55:44,137 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:55:44,137 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:55:44,137 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:44,138 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:55:44,140 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:55:44,140 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:44,140 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:55:44,140 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:44,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:44,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:44,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:44,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:44,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:44,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:44,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:44,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:44,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:44,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:44,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:55:44,144 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:55:44,144 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:55:44,144 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:55:44,144 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:55:44,144 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:55:44,144 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:55:44,150 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:44,150 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:44,150 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:44,150 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:44,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:44,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:44,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:44,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:44,151 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:44,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:44,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:44,151 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:44,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:44,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:44,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:44,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:44,153 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:44,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:44,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:44,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:44,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:44,155 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:44,155 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:44,155 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:44,155 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:44,155 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:44,155 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:44,155 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:55:44,155 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:55:44,155 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:55:44,155 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:44,155 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:44,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:44,155 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:44,155 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:44,155 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:44,155 DEBUG [Thread-2900] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:55:44,156 DEBUG [Thread-2900] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:55:44,156 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:44,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:44,256 DEBUG [Thread-2900] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:55:44,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:55:44,258 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:55:44,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:55:44,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:55:44,259 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:55:44,259 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:44,259 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:55:44,259 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:55:44,259 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:44,259 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:44,260 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:44,260 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:44,261 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:44,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:44,261 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:44,261 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:55:44,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:44,261 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:44,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:55:44,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:44,261 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:55:44,267 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:55:44,267 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=160 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp/B/f503ad2c94fd41cbb1fc9a2e5502158d 2023-04-25 02:55:44,267 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:55:44,267 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:55:44,267 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:44,267 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:55:44,272 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:55:44,272 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:55:44,272 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. 2023-04-25 02:55:44,272 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. started... 2023-04-25 02:55:44,272 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. as already flushing 2023-04-25 02:55:44,272 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. 2023-04-25 02:55:44,272 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:55:44,272 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:55:44,272 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:44,272 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:55:44,272 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:55:44,274 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:44,274 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:44,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:44,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:44,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:44,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:44,275 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:44,275 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:55:44,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:44,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:44,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:44,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:44,276 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:55:44,276 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@141c6e5e[Count = 0] remaining members to acquire global barrier 2023-04-25 02:55:44,276 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:55:44,276 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:44,278 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:44,278 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:44,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:44,278 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:55:44,278 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:44,278 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:55:44,278 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:55:44,279 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:55:44,280 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:44,280 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:55:44,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:44,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:44,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:44,280 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:44,280 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:55:44,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:44,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:44,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:44,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:44,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:44,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:44,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:44,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:55:44,283 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:55:44,283 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:55:44,283 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:55:44,283 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:55:44,283 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:55:44,283 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:55:44,287 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:44,287 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:44,287 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:44,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:44,287 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:44,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:44,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:44,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:44,288 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:44,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:44,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:44,288 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:44,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:44,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:44,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:44,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:44,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:44,289 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:44,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:44,291 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:44,291 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:44,291 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:44,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:44,291 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:44,291 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:44,291 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:44,291 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:44,292 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:44,292 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:44,292 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:55:44,292 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:44,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:44,292 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:55:44,292 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:44,292 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:44,292 DEBUG [Thread-2900] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:55:44,292 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:55:44,292 DEBUG [Thread-2900] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:55:44,392 DEBUG [Thread-2900] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:55:44,393 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:55:44,395 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:55:44,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:55:44,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:55:44,396 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:55:44,396 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:44,396 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:55:44,396 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:55:44,396 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:44,396 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:44,403 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:44,403 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:44,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:44,403 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:44,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:44,404 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:44,404 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:55:44,404 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:44,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:55:44,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:44,404 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:55:44,404 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:55:44,404 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:55:44,404 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:55:44,405 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:44,405 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:55:44,405 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:55:44,405 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. 2023-04-25 02:55:44,405 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. started... 2023-04-25 02:55:44,405 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:55:44,405 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. as already flushing 2023-04-25 02:55:44,405 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. 2023-04-25 02:55:44,405 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:55:44,405 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:55:44,405 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:44,405 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:55:44,405 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:55:44,406 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:44,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:44,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:44,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:44,407 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:44,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:44,407 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:44,407 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:55:44,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:44,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:44,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:44,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:44,408 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:55:44,408 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7d591008[Count = 0] remaining members to acquire global barrier 2023-04-25 02:55:44,408 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:55:44,408 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:44,409 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:44,409 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:44,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:44,410 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:55:44,410 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:55:44,410 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:55:44,410 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:44,410 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:55:44,411 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:55:44,411 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:44,411 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:44,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:44,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:44,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:44,411 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:55:44,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:44,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:44,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:44,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:44,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:44,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:44,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:44,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:55:44,413 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:55:44,413 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:55:44,413 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:55:44,413 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:55:44,413 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:55:44,413 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:55:44,416 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:44,416 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:44,416 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:44,416 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:44,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:44,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:44,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:44,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:44,416 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:44,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:44,416 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:44,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:44,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:44,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:44,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:44,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:44,417 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:44,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:44,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:44,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:44,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:44,419 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:44,419 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:44,419 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:44,419 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:44,419 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:55:44,419 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:55:44,419 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:44,419 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:44,419 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:55:44,419 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:44,419 DEBUG [Thread-2900] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:55:44,419 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:44,419 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:44,420 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:44,419 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:44,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:44,419 DEBUG [Thread-2900] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:55:44,420 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:44,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:44,520 DEBUG [Thread-2900] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:55:44,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:55:44,522 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:55:44,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:55:44,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:55:44,523 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:55:44,523 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:44,523 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:55:44,523 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:55:44,524 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:44,524 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:44,525 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:44,525 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:44,525 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:44,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:44,525 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:44,525 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:55:44,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:44,526 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:44,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:55:44,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:44,526 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:55:44,526 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:55:44,526 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:55:44,526 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:55:44,526 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:44,526 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:55:44,527 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:55:44,527 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:55:44,527 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. 2023-04-25 02:55:44,527 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. started... 2023-04-25 02:55:44,527 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. as already flushing 2023-04-25 02:55:44,527 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. 2023-04-25 02:55:44,527 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:55:44,527 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:55:44,527 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:44,527 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:55:44,527 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:55:44,528 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:44,528 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:44,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:44,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:44,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:44,529 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:44,529 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:55:44,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:44,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:44,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:44,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:44,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:44,530 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:55:44,530 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4551c0bb[Count = 0] remaining members to acquire global barrier 2023-04-25 02:55:44,530 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:55:44,530 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:44,532 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:44,532 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:44,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:44,532 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:55:44,532 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:55:44,532 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:55:44,532 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:44,532 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:55:44,533 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:44,533 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:55:44,534 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:44,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:44,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:44,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:44,534 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:55:44,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:44,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:44,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:44,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:44,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:44,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:44,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:44,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:55:44,546 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:55:44,546 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:55:44,546 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:55:44,546 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:55:44,546 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:55:44,546 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:55:44,547 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:44,547 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:44,547 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:44,547 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:44,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:44,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:44,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:44,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:44,548 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:44,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:44,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:44,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:44,548 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:44,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:44,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:44,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:44,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:44,549 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:44,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:44,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:44,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:44,558 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:44,558 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:44,558 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:44,558 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:55:44,558 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:55:44,558 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:44,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:44,558 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:44,558 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:44,558 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:44,558 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:44,558 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:44,558 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:44,559 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:44,559 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:44,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:44,558 DEBUG [Thread-2900] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:55:44,559 DEBUG [Thread-2900] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:55:44,578 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:44,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59114 deadline: 1682391404568, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:44,579 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:44,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59156 deadline: 1682391404575, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:44,580 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:44,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59116 deadline: 1682391404576, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:44,597 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:44,597 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59162 deadline: 1682391404590, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:44,659 DEBUG [Thread-2900] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:55:44,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:55:44,661 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:55:44,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:55:44,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:55:44,662 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:55:44,662 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:44,663 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:55:44,663 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:55:44,663 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:44,663 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:44,664 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:44,664 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:44,664 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:44,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:44,665 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:44,665 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:55:44,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:44,665 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:44,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:55:44,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:44,666 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:55:44,666 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:55:44,666 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:55:44,666 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:55:44,666 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:44,666 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:55:44,666 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:55:44,666 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:55:44,666 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. 2023-04-25 02:55:44,666 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. started... 2023-04-25 02:55:44,666 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. as already flushing 2023-04-25 02:55:44,666 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. 2023-04-25 02:55:44,666 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:55:44,666 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:55:44,667 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:44,667 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:55:44,667 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:55:44,668 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:44,668 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:44,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:44,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:44,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:44,668 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:44,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:44,669 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:55:44,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:44,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:44,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:44,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:44,670 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:55:44,670 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@10f86b84[Count = 0] remaining members to acquire global barrier 2023-04-25 02:55:44,670 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:55:44,670 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:44,671 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:44,671 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:44,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:44,671 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:55:44,671 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:55:44,671 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:55:44,672 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:44,672 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:55:44,674 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:55:44,674 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:44,674 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:55:44,674 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:44,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:44,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:44,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:44,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:44,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:44,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:44,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:44,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:44,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:44,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:44,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:55:44,676 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:55:44,676 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:55:44,676 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:55:44,676 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:55:44,677 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:55:44,677 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:55:44,678 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:44,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:44,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:44,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:44,678 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:44,678 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:44,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:44,678 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:44,678 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:44,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:44,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:44,679 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:44,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:44,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:44,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:44,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:44,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:44,681 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:44,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:44,682 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=160 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp/C/163ff9da8ebe4a6a8b338efd1173fdd4 2023-04-25 02:55:44,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:44,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:44,684 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:44,684 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:44,684 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:44,684 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:44,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:44,684 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:44,684 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:44,684 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:55:44,684 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:55:44,684 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:44,685 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:44,685 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:44,685 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:44,685 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:44,685 DEBUG [Thread-2900] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:55:44,685 DEBUG [Thread-2900] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:55:44,686 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:44,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:44,687 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:55:44,687 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp/A/fbe6297e3cf8405cad592ff83411a75d as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/fbe6297e3cf8405cad592ff83411a75d 2023-04-25 02:55:44,698 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/fbe6297e3cf8405cad592ff83411a75d, entries=200, sequenceid=160, filesize=14.1 K 2023-04-25 02:55:44,699 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp/B/f503ad2c94fd41cbb1fc9a2e5502158d as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/f503ad2c94fd41cbb1fc9a2e5502158d 2023-04-25 02:55:44,704 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/f503ad2c94fd41cbb1fc9a2e5502158d, entries=150, sequenceid=160, filesize=11.8 K 2023-04-25 02:55:44,705 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp/C/163ff9da8ebe4a6a8b338efd1173fdd4 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/163ff9da8ebe4a6a8b338efd1173fdd4 2023-04-25 02:55:44,708 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/163ff9da8ebe4a6a8b338efd1173fdd4, entries=150, sequenceid=160, filesize=11.8 K 2023-04-25 02:55:44,709 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 5f2c97e00c770d329321cff2fbd6848c in 1309ms, sequenceid=160, compaction requested=true 2023-04-25 02:55:44,709 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 5f2c97e00c770d329321cff2fbd6848c: 2023-04-25 02:55:44,709 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 02:55:44,709 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-25 02:55:44,709 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-25 02:55:44,709 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 02:55:44,709 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 02:55:44,711 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 88335 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-04-25 02:55:44,711 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 83605 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-04-25 02:55:44,711 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HStore(1912): 5f2c97e00c770d329321cff2fbd6848c/A is initiating minor compaction (all files) 2023-04-25 02:55:44,711 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HStore(1912): 5f2c97e00c770d329321cff2fbd6848c/B is initiating minor compaction (all files) 2023-04-25 02:55:44,711 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 5f2c97e00c770d329321cff2fbd6848c/A in TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. 2023-04-25 02:55:44,711 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 5f2c97e00c770d329321cff2fbd6848c/B in TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. 2023-04-25 02:55:44,712 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/cac2fa2cedc34beabbd5ee13b997478c, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/23e4830258f74dbba9a73b11c9d5f826, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/329c77bde63b4a60bcaba2a4365c7d99, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/1713671593824abd8ab8ddb68b32e1b7, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/25bb38aa1ffa4896a8c2765283fe8746, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/4b83b1d55379429ba95b44494cb20aea, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/fbe6297e3cf8405cad592ff83411a75d] into tmpdir=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp, totalSize=86.3 K 2023-04-25 02:55:44,712 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/1fc484b1f6d44b9aa4d98ea4c00115bc, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/2410eb188561495fa42936ef537ce3f9, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/1e555b288f224498a7ca167aaa30d2ad, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/c38c46225b7749fb978d49bdf27c4e26, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/041e58301c0042749998a15ae582afbb, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/6439852b16434808a0763062e981e154, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/f503ad2c94fd41cbb1fc9a2e5502158d] into tmpdir=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp, totalSize=81.6 K 2023-04-25 02:55:44,712 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting cac2fa2cedc34beabbd5ee13b997478c, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=60, earliestPutTs=1682391337978 2023-04-25 02:55:44,712 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting 1fc484b1f6d44b9aa4d98ea4c00115bc, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=60, earliestPutTs=1682391337978 2023-04-25 02:55:44,712 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting 23e4830258f74dbba9a73b11c9d5f826, keycount=200, bloomtype=ROW, size=13.9 K, encoding=NONE, compression=NONE, seqNum=80, earliestPutTs=1682391338310 2023-04-25 02:55:44,713 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting 2410eb188561495fa42936ef537ce3f9, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=80, earliestPutTs=1682391338323 2023-04-25 02:55:44,713 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting 329c77bde63b4a60bcaba2a4365c7d99, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=95, earliestPutTs=1682391338452 2023-04-25 02:55:44,713 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting 1e555b288f224498a7ca167aaa30d2ad, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=95, earliestPutTs=1682391338452 2023-04-25 02:55:44,713 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting 1713671593824abd8ab8ddb68b32e1b7, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=115, earliestPutTs=1682391339602 2023-04-25 02:55:44,714 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting c38c46225b7749fb978d49bdf27c4e26, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=115, earliestPutTs=1682391339602 2023-04-25 02:55:44,714 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting 25bb38aa1ffa4896a8c2765283fe8746, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=127, earliestPutTs=1682391340751 2023-04-25 02:55:44,714 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting 041e58301c0042749998a15ae582afbb, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=127, earliestPutTs=1682391340751 2023-04-25 02:55:44,714 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting 4b83b1d55379429ba95b44494cb20aea, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=147, earliestPutTs=1682391341138 2023-04-25 02:55:44,715 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting fbe6297e3cf8405cad592ff83411a75d, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=160, earliestPutTs=1682391343285 2023-04-25 02:55:44,715 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting 6439852b16434808a0763062e981e154, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=147, earliestPutTs=1682391341138 2023-04-25 02:55:44,715 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting f503ad2c94fd41cbb1fc9a2e5502158d, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=160, earliestPutTs=1682391343285 2023-04-25 02:55:44,736 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] throttle.PressureAwareThroughputController(145): 5f2c97e00c770d329321cff2fbd6848c#A#compaction#397 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-25 02:55:44,736 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] throttle.PressureAwareThroughputController(145): 5f2c97e00c770d329321cff2fbd6848c#B#compaction#396 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-25 02:55:44,774 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp/B/902a85c141fe4d3683f8ac3de6c5e331 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/902a85c141fe4d3683f8ac3de6c5e331 2023-04-25 02:55:44,779 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 5f2c97e00c770d329321cff2fbd6848c/B of 5f2c97e00c770d329321cff2fbd6848c into 902a85c141fe4d3683f8ac3de6c5e331(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 02:55:44,779 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5f2c97e00c770d329321cff2fbd6848c: 2023-04-25 02:55:44,779 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c., storeName=5f2c97e00c770d329321cff2fbd6848c/B, priority=9, startTime=1682391344709; duration=0sec 2023-04-25 02:55:44,779 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 02:55:44,779 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-25 02:55:44,781 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 83605 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-04-25 02:55:44,781 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HStore(1912): 5f2c97e00c770d329321cff2fbd6848c/C is initiating minor compaction (all files) 2023-04-25 02:55:44,781 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 5f2c97e00c770d329321cff2fbd6848c/C in TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. 2023-04-25 02:55:44,781 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/c39af132f78942f5a05423334d8ab1b6, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/a537c55d4a7b4b2990fa3405e4faa39b, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/05f9721c3c724ab0a2ac4f8dcd81f1a4, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/14a6f6a7152b44e48dc44f684cf6f6d6, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/bc783b0c9dc54965a05b5ae9c1b9a730, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/67b65249b4b04c09bdb0a8f5619d5d6e, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/163ff9da8ebe4a6a8b338efd1173fdd4] into tmpdir=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp, totalSize=81.6 K 2023-04-25 02:55:44,782 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting c39af132f78942f5a05423334d8ab1b6, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=60, earliestPutTs=1682391337978 2023-04-25 02:55:44,782 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting a537c55d4a7b4b2990fa3405e4faa39b, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=80, earliestPutTs=1682391338323 2023-04-25 02:55:44,782 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting 05f9721c3c724ab0a2ac4f8dcd81f1a4, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=95, earliestPutTs=1682391338452 2023-04-25 02:55:44,783 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting 14a6f6a7152b44e48dc44f684cf6f6d6, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=115, earliestPutTs=1682391339602 2023-04-25 02:55:44,783 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting bc783b0c9dc54965a05b5ae9c1b9a730, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=127, earliestPutTs=1682391340751 2023-04-25 02:55:44,783 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting 67b65249b4b04c09bdb0a8f5619d5d6e, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=147, earliestPutTs=1682391341138 2023-04-25 02:55:44,784 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting 163ff9da8ebe4a6a8b338efd1173fdd4, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=160, earliestPutTs=1682391343285 2023-04-25 02:55:44,785 DEBUG [Thread-2900] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:55:44,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:55:44,788 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:55:44,812 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] throttle.PressureAwareThroughputController(145): 5f2c97e00c770d329321cff2fbd6848c#C#compaction#398 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-25 02:55:44,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:55:44,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:55:44,824 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:55:44,824 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:44,825 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:55:44,825 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:55:44,825 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:44,825 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:44,826 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:44,826 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:44,826 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:44,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:44,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:44,827 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:44,827 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:55:44,827 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:44,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:55:44,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:44,828 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:55:44,832 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:55:44,832 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:55:44,832 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:55:44,832 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:44,832 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:55:44,836 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:55:44,836 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:55:44,836 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. 2023-04-25 02:55:44,836 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. started... 2023-04-25 02:55:44,836 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5f2c97e00c770d329321cff2fbd6848c 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-25 02:55:44,836 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5f2c97e00c770d329321cff2fbd6848c, store=A 2023-04-25 02:55:44,837 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:55:44,837 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5f2c97e00c770d329321cff2fbd6848c, store=B 2023-04-25 02:55:44,837 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:55:44,837 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5f2c97e00c770d329321cff2fbd6848c, store=C 2023-04-25 02:55:44,837 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:55:44,838 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp/A/3aa6211c6eca46298db69d5be804516a as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/3aa6211c6eca46298db69d5be804516a 2023-04-25 02:55:44,848 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 5f2c97e00c770d329321cff2fbd6848c/A of 5f2c97e00c770d329321cff2fbd6848c into 3aa6211c6eca46298db69d5be804516a(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 02:55:44,849 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5f2c97e00c770d329321cff2fbd6848c: 2023-04-25 02:55:44,849 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c., storeName=5f2c97e00c770d329321cff2fbd6848c/A, priority=9, startTime=1682391344709; duration=0sec 2023-04-25 02:55:44,849 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 02:55:44,871 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp/C/8987c4e8b3f3427d8f4ea0faeb912a71 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/8987c4e8b3f3427d8f4ea0faeb912a71 2023-04-25 02:55:44,882 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 5f2c97e00c770d329321cff2fbd6848c/C of 5f2c97e00c770d329321cff2fbd6848c into 8987c4e8b3f3427d8f4ea0faeb912a71(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 02:55:44,882 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5f2c97e00c770d329321cff2fbd6848c: 2023-04-25 02:55:44,882 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c., storeName=5f2c97e00c770d329321cff2fbd6848c/C, priority=9, startTime=1682391344709; duration=0sec 2023-04-25 02:55:44,882 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 02:55:44,883 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=180 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp/A/a80e49d868b64a3d9220fb8178b5d6a7 2023-04-25 02:55:44,902 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=180 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp/B/925ee47c2491406cad64a1714667faa7 2023-04-25 02:55:45,300 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. as already flushing 2023-04-25 02:55:45,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(9158): Flush requested on 5f2c97e00c770d329321cff2fbd6848c 2023-04-25 02:55:45,338 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=180 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp/C/ed9b1387563f4ed6a0e030fbcabd8c91 2023-04-25 02:55:45,343 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp/A/a80e49d868b64a3d9220fb8178b5d6a7 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/a80e49d868b64a3d9220fb8178b5d6a7 2023-04-25 02:55:45,348 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/a80e49d868b64a3d9220fb8178b5d6a7, entries=150, sequenceid=180, filesize=11.8 K 2023-04-25 02:55:45,349 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp/B/925ee47c2491406cad64a1714667faa7 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/925ee47c2491406cad64a1714667faa7 2023-04-25 02:55:45,376 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/925ee47c2491406cad64a1714667faa7, entries=150, sequenceid=180, filesize=11.8 K 2023-04-25 02:55:45,377 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp/C/ed9b1387563f4ed6a0e030fbcabd8c91 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/ed9b1387563f4ed6a0e030fbcabd8c91 2023-04-25 02:55:45,383 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:45,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59136 deadline: 1682391405376, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:45,384 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/ed9b1387563f4ed6a0e030fbcabd8c91, entries=150, sequenceid=180, filesize=11.8 K 2023-04-25 02:55:45,385 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 5f2c97e00c770d329321cff2fbd6848c in 549ms, sequenceid=180, compaction requested=false 2023-04-25 02:55:45,385 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5f2c97e00c770d329321cff2fbd6848c: 2023-04-25 02:55:45,385 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. 2023-04-25 02:55:45,385 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:55:45,385 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:55:45,385 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:45,388 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:55:45,388 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:55:45,391 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:45,391 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:45,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:45,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:45,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:45,391 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:45,391 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:55:45,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:45,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:45,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:45,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:45,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:45,393 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:55:45,393 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6761895b[Count = 0] remaining members to acquire global barrier 2023-04-25 02:55:45,393 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:55:45,393 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:45,394 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:45,394 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:45,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:45,394 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:55:45,394 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:55:45,394 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:55:45,397 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:45,397 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:55:45,397 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:45,397 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:55:45,398 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:45,398 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:55:45,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:45,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:45,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:45,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:45,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:45,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:45,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:45,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:45,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:45,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:45,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:55:45,401 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:55:45,401 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:55:45,401 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:55:45,402 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:55:45,402 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:55:45,402 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:55:45,403 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:45,403 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:45,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:45,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:45,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:45,403 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:45,403 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:45,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:45,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:45,403 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:45,404 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:45,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:45,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:45,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:45,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:45,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:45,407 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:45,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:45,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:45,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:45,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:45,410 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:45,410 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:45,410 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:45,410 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:45,410 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:45,410 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:45,410 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:45,410 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:45,410 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:45,410 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:45,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:45,411 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:45,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:45,411 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:45,411 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:55:45,411 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:55:45,412 DEBUG [Thread-2900] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:55:45,412 DEBUG [Thread-2900] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:55:45,412 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:55:45,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(9158): Flush requested on 5f2c97e00c770d329321cff2fbd6848c 2023-04-25 02:55:45,488 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 5f2c97e00c770d329321cff2fbd6848c 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-25 02:55:45,488 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5f2c97e00c770d329321cff2fbd6848c, store=A 2023-04-25 02:55:45,489 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:55:45,489 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5f2c97e00c770d329321cff2fbd6848c, store=B 2023-04-25 02:55:45,489 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:55:45,489 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5f2c97e00c770d329321cff2fbd6848c, store=C 2023-04-25 02:55:45,489 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:55:45,516 DEBUG [Thread-2900] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:55:45,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:55:45,528 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:55:45,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:55:45,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:55:45,536 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:55:45,536 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:45,536 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:55:45,536 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:55:45,537 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:45,537 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:45,540 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:45,540 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:45,540 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:45,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:45,541 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:45,541 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:55:45,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:45,545 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:45,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:55:45,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:45,546 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:55:45,550 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:55:45,550 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:55:45,550 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:55:45,550 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:45,550 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:55:45,561 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:55:45,561 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:55:45,562 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. 2023-04-25 02:55:45,565 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. started... 2023-04-25 02:55:45,565 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. as already flushing 2023-04-25 02:55:45,565 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. 2023-04-25 02:55:45,565 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:55:45,565 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:55:45,565 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:45,565 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:55:45,565 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:55:45,567 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:45,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:45,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:45,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:45,567 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:45,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:45,568 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:45,568 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:55:45,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:45,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:45,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:45,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:45,570 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:55:45,570 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@520d0cef[Count = 0] remaining members to acquire global barrier 2023-04-25 02:55:45,570 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:55:45,570 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:45,571 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:45,571 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:55:45,571 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:45,571 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:45,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:45,572 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:55:45,572 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:55:45,572 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:55:45,576 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:45,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:45,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:45,576 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:55:45,576 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:45,576 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:55:45,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:45,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:45,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:45,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:45,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:45,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:45,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:45,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:45,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:55:45,581 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:55:45,581 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:55:45,581 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:55:45,581 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:55:45,582 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:55:45,582 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:55:45,583 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:45,583 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:45,583 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:45,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:45,583 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:45,584 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:45,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:45,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:45,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:45,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:45,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:45,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:45,586 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:45,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:45,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:45,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:45,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:45,590 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:45,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:45,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:45,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:45,595 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:45,595 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:45,595 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:45,595 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:45,595 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:45,595 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:45,595 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:45,595 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:45,595 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:45,595 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:45,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:45,595 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:45,595 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:55:45,595 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:55:45,596 DEBUG [Thread-2900] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:55:45,596 DEBUG [Thread-2900] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:55:45,596 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:45,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:45,608 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:45,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59136 deadline: 1682391405596, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:45,608 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:45,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59156 deadline: 1682391405598, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:45,608 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:45,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59114 deadline: 1682391405598, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:45,610 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:45,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59116 deadline: 1682391405600, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:45,619 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:45,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59162 deadline: 1682391405611, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:45,704 DEBUG [Thread-2900] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:55:45,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:55:45,716 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:45,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59116 deadline: 1682391405712, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:45,717 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:45,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59156 deadline: 1682391405712, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:45,718 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:45,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59136 deadline: 1682391405712, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:45,718 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:45,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59114 deadline: 1682391405712, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:45,720 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:55:45,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:55:45,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:55:45,728 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:55:45,728 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:45,728 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:55:45,728 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:55:45,728 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:45,728 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:45,732 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:45,732 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:45,732 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:45,732 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:55:45,733 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:45,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:45,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:45,734 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:45,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:55:45,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:45,734 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:55:45,735 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:55:45,735 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:55:45,735 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:55:45,735 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:45,735 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:55:45,749 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:55:45,750 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:55:45,750 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. 2023-04-25 02:55:45,750 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. started... 2023-04-25 02:55:45,750 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. as already flushing 2023-04-25 02:55:45,750 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. 2023-04-25 02:55:45,750 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:55:45,750 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:55:45,750 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:45,750 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:55:45,750 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:55:45,752 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:45,753 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:45,753 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:55:45,761 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:45,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:45,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:45,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:45,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:45,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:45,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:45,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:45,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:45,765 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:55:45,765 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@19925a0d[Count = 0] remaining members to acquire global barrier 2023-04-25 02:55:45,765 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:55:45,765 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:45,766 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:45,766 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:55:45,767 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:45,767 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:45,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:45,767 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:55:45,767 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:55:45,767 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:55:45,769 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:55:45,769 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:45,769 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:55:45,773 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:45,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:45,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:45,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:45,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:45,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:45,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:45,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:45,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:45,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:45,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:45,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:55:45,783 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:55:45,783 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:55:45,783 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:55:45,783 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:55:45,783 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:55:45,783 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:55:45,788 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:45,788 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:45,788 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:45,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:45,788 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:45,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:45,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:45,789 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:45,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:45,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:45,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:45,790 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:45,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:45,792 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:45,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:45,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:45,800 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:45,800 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:45,800 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:45,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:45,801 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:45,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:45,802 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:45,802 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:45,802 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:45,802 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:45,802 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:45,803 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:45,803 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:45,803 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:45,803 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:55:45,803 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:55:45,803 DEBUG [Thread-2900] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:55:45,803 DEBUG [Thread-2900] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:55:45,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:45,804 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:55:45,903 DEBUG [Thread-2900] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:55:45,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:55:45,906 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:55:45,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:55:45,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:55:45,910 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:55:45,910 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:45,910 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:55:45,910 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:55:45,910 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:45,910 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:45,911 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:45,912 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:45,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:45,912 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:45,912 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:45,912 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:55:45,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:45,913 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:45,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:55:45,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:45,913 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:55:45,925 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:55:45,925 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:55:45,925 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:55:45,925 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:45,925 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:55:45,926 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:45,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59156 deadline: 1682391405918, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:45,927 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:45,927 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59136 deadline: 1682391405920, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:45,927 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:45,927 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59114 deadline: 1682391405920, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:45,928 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:45,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59116 deadline: 1682391405924, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:45,932 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:55:45,932 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:55:45,933 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. 2023-04-25 02:55:45,933 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. started... 2023-04-25 02:55:45,933 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. as already flushing 2023-04-25 02:55:45,933 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. 2023-04-25 02:55:45,933 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=196 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp/A/e112f6498cff4b439b8357b0c8ed59b0 2023-04-25 02:55:45,933 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:55:45,933 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:55:45,933 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:45,933 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:55:45,933 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:55:45,935 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:45,935 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:45,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:45,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:45,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:45,935 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:45,936 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:55:45,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:45,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:45,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:45,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:45,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:45,938 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:55:45,938 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4a65b6f0[Count = 0] remaining members to acquire global barrier 2023-04-25 02:55:45,938 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:55:45,938 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:45,939 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:45,939 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:45,939 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:55:45,939 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:45,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:45,939 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:55:45,939 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:55:45,939 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:55:45,954 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:45,955 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:55:45,955 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:45,955 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:55:45,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:45,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:45,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:45,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:45,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:45,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:45,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:45,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:45,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:45,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:45,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:55:45,969 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:55:45,969 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:55:45,969 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:55:45,969 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:55:45,969 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:55:45,969 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:55:45,972 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:45,972 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:45,972 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:45,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:45,973 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:45,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:45,973 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:45,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:45,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:45,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:45,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:45,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:45,974 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:45,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:45,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:45,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:45,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:45,975 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:45,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:45,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:45,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:45,980 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:45,980 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:45,980 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:45,980 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:55:45,980 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:55:45,981 DEBUG [Thread-2900] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:55:45,981 DEBUG [Thread-2900] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:55:45,980 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:45,980 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:45,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:45,980 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:45,981 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:45,981 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:45,981 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:45,981 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:45,981 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:45,981 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:45,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:46,081 DEBUG [Thread-2900] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:55:46,081 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:55:46,083 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:55:46,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:55:46,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:55:46,092 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:55:46,092 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:46,092 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:55:46,092 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:55:46,100 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:46,100 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:46,101 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:46,101 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:46,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:46,101 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:46,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:46,102 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:46,102 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:55:46,102 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:46,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:55:46,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:46,103 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:55:46,111 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:55:46,111 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:55:46,111 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:55:46,111 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:46,111 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:55:46,116 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:55:46,116 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:55:46,116 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. 2023-04-25 02:55:46,116 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. started... 2023-04-25 02:55:46,116 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. as already flushing 2023-04-25 02:55:46,116 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. 2023-04-25 02:55:46,116 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:55:46,116 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:55:46,116 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:46,116 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:55:46,116 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:55:46,119 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:46,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:46,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:46,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:46,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:46,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:46,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:46,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:46,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:46,121 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:55:46,121 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2125c5b6[Count = 0] remaining members to acquire global barrier 2023-04-25 02:55:46,121 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:55:46,121 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:46,122 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:46,122 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:55:46,124 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:46,125 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(162): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:46,125 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:46,125 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:55:46,125 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:55:46,125 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:55:46,125 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:55:46,126 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:46,126 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:55:46,126 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:46,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:46,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:46,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:46,126 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:55:46,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:46,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:46,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:46,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:46,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:46,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:46,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:46,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:55:46,129 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:55:46,129 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:55:46,129 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:55:46,129 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:55:46,129 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:55:46,129 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:55:46,131 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:46,131 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:46,131 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:46,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:46,131 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:46,131 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:46,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:46,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:46,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:46,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:46,132 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:46,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:46,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:46,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:46,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:46,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:46,135 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:46,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:46,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:46,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:46,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:46,148 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:46,148 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:46,148 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:46,148 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:55:46,148 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:55:46,148 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:46,148 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:46,148 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:46,148 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:46,148 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:46,148 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:46,148 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:46,148 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:55:46,148 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:46,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:46,148 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:46,148 DEBUG [Thread-2900] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:55:46,149 DEBUG [Thread-2900] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:55:46,149 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:46,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:46,239 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:46,240 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:46,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59136 deadline: 1682391406230, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:46,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59156 deadline: 1682391406231, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:46,240 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:46,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59114 deadline: 1682391406231, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:46,241 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:46,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59116 deadline: 1682391406232, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:46,249 DEBUG [Thread-2900] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:55:46,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:55:46,254 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:55:46,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:55:46,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:55:46,256 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:55:46,256 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:46,256 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:55:46,256 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:55:46,256 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:46,256 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:46,258 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:46,258 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:46,258 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:46,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:46,258 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:46,258 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:55:46,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:46,258 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:46,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:55:46,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:46,259 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:55:46,264 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:55:46,264 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:55:46,264 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:55:46,264 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:46,264 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:55:46,276 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:55:46,276 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:55:46,279 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. 2023-04-25 02:55:46,279 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. started... 2023-04-25 02:55:46,279 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. as already flushing 2023-04-25 02:55:46,279 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. 2023-04-25 02:55:46,279 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:55:46,279 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:55:46,279 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:46,279 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:55:46,279 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:55:46,283 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:46,283 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:46,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:46,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:46,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:46,284 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:46,284 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:55:46,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:46,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:46,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:46,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:46,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:46,285 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:55:46,285 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1bf5df09[Count = 0] remaining members to acquire global barrier 2023-04-25 02:55:46,285 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:55:46,285 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:46,287 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:46,287 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:46,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:46,288 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:55:46,288 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:55:46,288 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:55:46,289 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:46,289 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:55:46,289 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:46,289 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:55:46,289 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:55:46,289 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:55:46,289 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:55:46,289 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:55:46,289 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:55:46,289 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:55:46,289 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:55:46,291 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:46,291 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:46,291 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:46,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:46,291 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:46,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:46,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:46,292 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:46,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:46,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:46,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:46,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:46,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:46,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:46,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:46,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:46,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:46,294 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:46,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:46,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:46,295 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:46,297 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:46,297 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:46,297 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:46,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:46,297 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:46,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:46,298 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:46,298 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:46,298 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:46,298 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:46,298 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:46,298 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:46,298 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:46,298 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:46,298 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:55:46,298 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:55:46,299 DEBUG [Thread-2900] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:55:46,299 DEBUG [Thread-2900] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:55:46,399 DEBUG [Thread-2900] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:55:46,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:55:46,401 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:55:46,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:55:46,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:55:46,402 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:55:46,402 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:46,402 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:55:46,402 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:55:46,403 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:46,403 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:46,405 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:46,405 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:46,406 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:46,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:46,406 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:46,406 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:55:46,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:46,406 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:46,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:55:46,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:46,407 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:55:46,407 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:55:46,407 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:55:46,407 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:55:46,407 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:46,407 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:55:46,408 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:55:46,408 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:55:46,408 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. 2023-04-25 02:55:46,408 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. started... 2023-04-25 02:55:46,408 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. as already flushing 2023-04-25 02:55:46,408 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. 2023-04-25 02:55:46,408 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:55:46,408 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:55:46,408 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:46,408 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:55:46,408 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:55:46,410 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:46,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:46,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:46,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:46,410 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:46,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:46,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:46,410 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:46,411 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:55:46,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:46,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:46,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:46,412 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:55:46,412 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@103cbc48[Count = 0] remaining members to acquire global barrier 2023-04-25 02:55:46,412 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:55:46,412 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:46,412 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=196 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp/B/1e20a7b405274c43bab50af4c413fc96 2023-04-25 02:55:46,413 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:46,413 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:55:46,413 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:46,413 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:46,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:46,414 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:55:46,414 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:55:46,414 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:55:46,415 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:55:46,415 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:46,415 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:55:46,416 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:46,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:46,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:46,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:46,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:46,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:46,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:46,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:46,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:46,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:46,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:46,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:55:46,422 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:55:46,422 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:55:46,422 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:55:46,422 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:55:46,422 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:55:46,422 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:55:46,427 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:46,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:46,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:46,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:46,427 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:46,427 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:46,427 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:46,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:46,428 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:46,428 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:46,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:46,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:46,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:46,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:46,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:46,430 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:46,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:46,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:46,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:46,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:46,432 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:46,432 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:46,433 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:46,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:46,433 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:46,433 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:46,433 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:46,433 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:55:46,433 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:55:46,433 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:46,433 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:46,433 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:46,433 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:46,433 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:46,433 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:46,433 DEBUG [Thread-2900] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:55:46,433 DEBUG [Thread-2900] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:55:46,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:46,434 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:55:46,439 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=196 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp/C/3928cbd922c14c2a8b0dd4c075905326 2023-04-25 02:55:46,447 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp/A/e112f6498cff4b439b8357b0c8ed59b0 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/e112f6498cff4b439b8357b0c8ed59b0 2023-04-25 02:55:46,454 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/e112f6498cff4b439b8357b0c8ed59b0, entries=200, sequenceid=196, filesize=14.1 K 2023-04-25 02:55:46,454 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp/B/1e20a7b405274c43bab50af4c413fc96 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/1e20a7b405274c43bab50af4c413fc96 2023-04-25 02:55:46,458 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/1e20a7b405274c43bab50af4c413fc96, entries=150, sequenceid=196, filesize=11.8 K 2023-04-25 02:55:46,459 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp/C/3928cbd922c14c2a8b0dd4c075905326 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/3928cbd922c14c2a8b0dd4c075905326 2023-04-25 02:55:46,471 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/3928cbd922c14c2a8b0dd4c075905326, entries=150, sequenceid=196, filesize=11.8 K 2023-04-25 02:55:46,472 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=107.34 KB/109920 for 5f2c97e00c770d329321cff2fbd6848c in 984ms, sequenceid=196, compaction requested=true 2023-04-25 02:55:46,472 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 5f2c97e00c770d329321cff2fbd6848c: 2023-04-25 02:55:46,472 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 02:55:46,472 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-25 02:55:46,472 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 02:55:46,472 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-25 02:55:46,472 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 02:55:46,473 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 38738 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-25 02:55:46,473 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36348 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-25 02:55:46,473 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HStore(1912): 5f2c97e00c770d329321cff2fbd6848c/A is initiating minor compaction (all files) 2023-04-25 02:55:46,474 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HStore(1912): 5f2c97e00c770d329321cff2fbd6848c/B is initiating minor compaction (all files) 2023-04-25 02:55:46,474 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 5f2c97e00c770d329321cff2fbd6848c/A in TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. 2023-04-25 02:55:46,474 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 5f2c97e00c770d329321cff2fbd6848c/B in TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. 2023-04-25 02:55:46,474 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/3aa6211c6eca46298db69d5be804516a, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/a80e49d868b64a3d9220fb8178b5d6a7, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/e112f6498cff4b439b8357b0c8ed59b0] into tmpdir=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp, totalSize=37.8 K 2023-04-25 02:55:46,474 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/902a85c141fe4d3683f8ac3de6c5e331, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/925ee47c2491406cad64a1714667faa7, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/1e20a7b405274c43bab50af4c413fc96] into tmpdir=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp, totalSize=35.5 K 2023-04-25 02:55:46,474 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting 3aa6211c6eca46298db69d5be804516a, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=160, earliestPutTs=1682391343285 2023-04-25 02:55:46,474 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting 902a85c141fe4d3683f8ac3de6c5e331, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=160, earliestPutTs=1682391343285 2023-04-25 02:55:46,474 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting a80e49d868b64a3d9220fb8178b5d6a7, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=180, earliestPutTs=1682391343423 2023-04-25 02:55:46,474 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting 925ee47c2491406cad64a1714667faa7, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=180, earliestPutTs=1682391343423 2023-04-25 02:55:46,475 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting e112f6498cff4b439b8357b0c8ed59b0, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=196, earliestPutTs=1682391345310 2023-04-25 02:55:46,475 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting 1e20a7b405274c43bab50af4c413fc96, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=196, earliestPutTs=1682391345310 2023-04-25 02:55:46,481 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] throttle.PressureAwareThroughputController(145): 5f2c97e00c770d329321cff2fbd6848c#B#compaction#405 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-25 02:55:46,489 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] throttle.PressureAwareThroughputController(145): 5f2c97e00c770d329321cff2fbd6848c#A#compaction#406 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-25 02:55:46,529 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp/B/932e44e415b14d86aced9cb57cabdef5 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/932e44e415b14d86aced9cb57cabdef5 2023-04-25 02:55:46,535 DEBUG [Thread-2900] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:55:46,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:55:46,540 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp/A/52e807f0534b46299f8e5373f67266ba as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/52e807f0534b46299f8e5373f67266ba 2023-04-25 02:55:46,541 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 5f2c97e00c770d329321cff2fbd6848c/B of 5f2c97e00c770d329321cff2fbd6848c into 932e44e415b14d86aced9cb57cabdef5(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 02:55:46,541 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5f2c97e00c770d329321cff2fbd6848c: 2023-04-25 02:55:46,541 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c., storeName=5f2c97e00c770d329321cff2fbd6848c/B, priority=13, startTime=1682391346472; duration=0sec 2023-04-25 02:55:46,541 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 02:55:46,541 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-25 02:55:46,544 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:55:46,547 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36348 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-25 02:55:46,547 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HStore(1912): 5f2c97e00c770d329321cff2fbd6848c/C is initiating minor compaction (all files) 2023-04-25 02:55:46,547 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 5f2c97e00c770d329321cff2fbd6848c/C in TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. 2023-04-25 02:55:46,547 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/8987c4e8b3f3427d8f4ea0faeb912a71, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/ed9b1387563f4ed6a0e030fbcabd8c91, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/3928cbd922c14c2a8b0dd4c075905326] into tmpdir=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp, totalSize=35.5 K 2023-04-25 02:55:46,548 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting 8987c4e8b3f3427d8f4ea0faeb912a71, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=160, earliestPutTs=1682391343285 2023-04-25 02:55:46,548 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting ed9b1387563f4ed6a0e030fbcabd8c91, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=180, earliestPutTs=1682391343423 2023-04-25 02:55:46,548 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting 3928cbd922c14c2a8b0dd4c075905326, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=196, earliestPutTs=1682391345310 2023-04-25 02:55:46,555 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:55:46,555 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:55:46,556 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:55:46,556 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:46,556 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:55:46,556 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:55:46,557 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:46,557 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:46,558 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:46,558 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:46,558 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:46,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:46,559 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:46,559 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:55:46,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:46,559 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:46,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:55:46,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:46,559 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:55:46,563 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 5f2c97e00c770d329321cff2fbd6848c/A of 5f2c97e00c770d329321cff2fbd6848c into 52e807f0534b46299f8e5373f67266ba(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 02:55:46,563 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5f2c97e00c770d329321cff2fbd6848c: 2023-04-25 02:55:46,563 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c., storeName=5f2c97e00c770d329321cff2fbd6848c/A, priority=13, startTime=1682391346472; duration=0sec 2023-04-25 02:55:46,564 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 02:55:46,566 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:55:46,567 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:55:46,568 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:55:46,568 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:46,568 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:55:46,585 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] throttle.PressureAwareThroughputController(145): 5f2c97e00c770d329321cff2fbd6848c#C#compaction#407 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-25 02:55:46,590 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:55:46,591 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. 2023-04-25 02:55:46,591 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:55:46,591 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. started... 2023-04-25 02:55:46,591 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5f2c97e00c770d329321cff2fbd6848c 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-25 02:55:46,604 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5f2c97e00c770d329321cff2fbd6848c, store=A 2023-04-25 02:55:46,604 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:55:46,604 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5f2c97e00c770d329321cff2fbd6848c, store=B 2023-04-25 02:55:46,604 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:55:46,604 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5f2c97e00c770d329321cff2fbd6848c, store=C 2023-04-25 02:55:46,604 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:55:46,668 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp/C/dc77376dad9d4835bdf1c9fa7fa07407 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/dc77376dad9d4835bdf1c9fa7fa07407 2023-04-25 02:55:46,676 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 5f2c97e00c770d329321cff2fbd6848c/C of 5f2c97e00c770d329321cff2fbd6848c into dc77376dad9d4835bdf1c9fa7fa07407(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 02:55:46,677 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5f2c97e00c770d329321cff2fbd6848c: 2023-04-25 02:55:46,677 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c., storeName=5f2c97e00c770d329321cff2fbd6848c/C, priority=13, startTime=1682391346472; duration=0sec 2023-04-25 02:55:46,677 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 02:55:46,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(9158): Flush requested on 5f2c97e00c770d329321cff2fbd6848c 2023-04-25 02:55:46,746 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. as already flushing 2023-04-25 02:55:46,784 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:46,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59136 deadline: 1682391406769, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:46,792 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:46,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59114 deadline: 1682391406784, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:46,792 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:46,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59116 deadline: 1682391406784, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:46,793 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:46,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59156 deadline: 1682391406785, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:46,888 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:46,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59136 deadline: 1682391406885, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:46,896 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:46,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59116 deadline: 1682391406893, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:46,897 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:46,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59156 deadline: 1682391406894, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:46,897 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:46,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59114 deadline: 1682391406895, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:47,059 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=217 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp/A/287d09e1d924469b9e6a75a83fb1dd38 2023-04-25 02:55:47,084 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=217 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp/B/dbb2305ba9404b9aa60db115f822326c 2023-04-25 02:55:47,093 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:47,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59136 deadline: 1682391407091, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:47,102 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:47,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59156 deadline: 1682391407098, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:47,102 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:47,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59114 deadline: 1682391407098, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:47,102 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:47,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59116 deadline: 1682391407100, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:47,400 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:47,401 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59136 deadline: 1682391407398, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:47,408 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:47,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59116 deadline: 1682391407404, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:47,409 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:47,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59156 deadline: 1682391407405, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:47,409 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:47,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59114 deadline: 1682391407406, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:47,517 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=217 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp/C/d53625636187492a9e6e3f4adabf357a 2023-04-25 02:55:47,521 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp/A/287d09e1d924469b9e6a75a83fb1dd38 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/287d09e1d924469b9e6a75a83fb1dd38 2023-04-25 02:55:47,525 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/287d09e1d924469b9e6a75a83fb1dd38, entries=150, sequenceid=217, filesize=11.8 K 2023-04-25 02:55:47,526 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp/B/dbb2305ba9404b9aa60db115f822326c as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/dbb2305ba9404b9aa60db115f822326c 2023-04-25 02:55:47,530 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/dbb2305ba9404b9aa60db115f822326c, entries=150, sequenceid=217, filesize=11.8 K 2023-04-25 02:55:47,530 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp/C/d53625636187492a9e6e3f4adabf357a as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/d53625636187492a9e6e3f4adabf357a 2023-04-25 02:55:47,534 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/d53625636187492a9e6e3f4adabf357a, entries=150, sequenceid=217, filesize=11.8 K 2023-04-25 02:55:47,535 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=73.80 KB/75570 for 5f2c97e00c770d329321cff2fbd6848c in 944ms, sequenceid=217, compaction requested=false 2023-04-25 02:55:47,535 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5f2c97e00c770d329321cff2fbd6848c: 2023-04-25 02:55:47,535 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. 2023-04-25 02:55:47,535 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:55:47,535 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:55:47,535 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:47,535 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:55:47,535 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:55:47,539 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:47,539 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:47,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:47,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:47,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:47,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:47,540 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:47,540 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:55:47,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:47,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:47,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:47,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:47,542 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:55:47,542 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@77895a89[Count = 0] remaining members to acquire global barrier 2023-04-25 02:55:47,543 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:55:47,543 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:47,546 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:47,546 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:47,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:47,546 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:55:47,546 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:55:47,546 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:55:47,546 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:47,546 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:55:47,547 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:55:47,547 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:47,547 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:47,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:47,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:47,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:47,547 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:55:47,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:47,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:47,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:47,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:47,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:47,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:47,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:47,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:55:47,551 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:55:47,551 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:55:47,551 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:55:47,551 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:55:47,552 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:55:47,552 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:55:47,553 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:47,553 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:47,553 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:47,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:47,553 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:47,553 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:47,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:47,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:47,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:47,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:47,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:47,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:47,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:47,555 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:47,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:47,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:47,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:47,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:47,557 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:47,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:47,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:47,559 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:47,559 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:47,559 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:47,560 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:47,560 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:47,559 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:47,560 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:47,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:47,560 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:47,560 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:47,560 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:47,560 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:47,560 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:55:47,560 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:55:47,560 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:47,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:47,560 DEBUG [Thread-2900] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:55:47,560 DEBUG [Thread-2900] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:55:47,638 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 5f2c97e00c770d329321cff2fbd6848c 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-25 02:55:47,639 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5f2c97e00c770d329321cff2fbd6848c, store=A 2023-04-25 02:55:47,639 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:55:47,639 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5f2c97e00c770d329321cff2fbd6848c, store=B 2023-04-25 02:55:47,639 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:55:47,639 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5f2c97e00c770d329321cff2fbd6848c, store=C 2023-04-25 02:55:47,639 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:55:47,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(9158): Flush requested on 5f2c97e00c770d329321cff2fbd6848c 2023-04-25 02:55:47,661 DEBUG [Thread-2900] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:55:47,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:55:47,663 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:55:47,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:55:47,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:55:47,664 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:55:47,664 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:47,664 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:55:47,664 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:55:47,665 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:47,665 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:47,667 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:47,667 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:47,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:47,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:47,668 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:47,668 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:47,668 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:47,668 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:55:47,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:55:47,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:47,669 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:55:47,669 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:55:47,669 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:55:47,669 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:55:47,669 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:47,669 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:55:47,669 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:55:47,669 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:55:47,669 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. 2023-04-25 02:55:47,669 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. started... 2023-04-25 02:55:47,669 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. as already flushing 2023-04-25 02:55:47,670 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. 2023-04-25 02:55:47,670 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:55:47,670 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:55:47,670 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:47,670 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:55:47,670 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:55:47,671 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:47,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:47,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:47,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:47,672 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:47,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:47,672 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:47,672 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:55:47,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:47,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:47,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:47,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:47,675 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:55:47,675 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@788e29bb[Count = 0] remaining members to acquire global barrier 2023-04-25 02:55:47,675 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:55:47,675 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:47,678 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:47,678 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:47,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:47,678 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:55:47,678 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:55:47,678 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:55:47,678 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:47,678 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:55:47,684 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:55:47,684 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:47,685 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:55:47,685 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:47,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:47,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:47,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:47,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:47,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:47,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:47,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:47,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:47,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:47,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:47,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:55:47,688 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:55:47,688 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:55:47,688 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:55:47,688 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:55:47,688 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:55:47,688 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:55:47,690 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:47,690 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:47,690 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:47,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:47,690 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:47,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:47,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:47,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:47,690 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:47,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:47,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:47,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:47,691 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:47,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:47,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:47,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:47,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:47,695 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:47,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:47,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:47,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:47,698 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:47,698 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:47,698 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:47,698 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:47,698 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:47,698 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:47,698 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:47,698 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:47,698 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:47,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:47,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:47,698 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:47,699 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:55:47,699 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:55:47,699 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:47,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:47,699 DEBUG [Thread-2900] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:55:47,699 DEBUG [Thread-2900] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:55:47,732 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:47,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59162 deadline: 1682391407729, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:47,799 DEBUG [Thread-2900] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:55:47,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:55:47,801 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:55:47,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:55:47,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:55:47,803 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:55:47,803 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:47,803 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:55:47,803 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:55:47,803 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:47,803 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:47,804 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:47,804 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:47,804 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:47,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:47,804 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:47,804 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:55:47,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:47,805 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:47,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:55:47,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:47,805 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:55:47,805 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:55:47,805 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:55:47,806 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:55:47,806 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:47,806 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:55:47,806 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:55:47,806 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:55:47,806 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. 2023-04-25 02:55:47,806 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. started... 2023-04-25 02:55:47,806 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. as already flushing 2023-04-25 02:55:47,806 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. 2023-04-25 02:55:47,806 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:55:47,806 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:55:47,806 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:47,806 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:55:47,806 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:55:47,809 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:47,809 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:47,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:47,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:47,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:47,809 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:47,809 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:55:47,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:47,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:47,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:47,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:47,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:47,810 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:55:47,810 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4db6bc34[Count = 0] remaining members to acquire global barrier 2023-04-25 02:55:47,810 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:55:47,810 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:47,811 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:47,811 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:47,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:47,812 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:55:47,812 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:55:47,812 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:55:47,813 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:47,813 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:55:47,814 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:47,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:47,814 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:55:47,814 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:47,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:47,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:47,814 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:55:47,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:47,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:47,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:47,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:47,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:47,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:47,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:47,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:55:47,816 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:55:47,816 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:55:47,816 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:55:47,816 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:55:47,816 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:55:47,816 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:55:47,817 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:47,818 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:47,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:47,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:47,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:47,818 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:47,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:47,818 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:47,818 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:47,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:47,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:47,818 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:47,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:47,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:47,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:47,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:47,819 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:47,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:47,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:47,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:47,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:47,822 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:47,823 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:47,823 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:47,823 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:47,823 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:47,823 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:55:47,823 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:55:47,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:47,823 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:55:47,823 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:47,823 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:47,823 DEBUG [Thread-2900] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:55:47,823 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:47,823 DEBUG [Thread-2900] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:55:47,823 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:47,823 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:47,823 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:47,823 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:47,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:47,840 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:47,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59162 deadline: 1682391407833, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:47,907 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:47,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59136 deadline: 1682391407903, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:47,913 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:47,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59114 deadline: 1682391407910, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:47,915 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:47,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59116 deadline: 1682391407912, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:47,917 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:47,917 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59156 deadline: 1682391407914, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:47,923 DEBUG [Thread-2900] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:55:47,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:55:47,925 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:55:47,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:55:47,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:55:47,927 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:55:47,927 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:47,927 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:55:47,927 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:55:47,927 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:47,927 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:47,929 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:47,929 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:47,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:47,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:47,930 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:47,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:55:47,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:47,930 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:55:47,930 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:47,930 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:55:47,930 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:55:47,930 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:55:47,930 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:47,930 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:55:47,931 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:47,931 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:55:47,931 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:55:47,931 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:55:47,932 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. 2023-04-25 02:55:47,932 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. started... 2023-04-25 02:55:47,932 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. as already flushing 2023-04-25 02:55:47,932 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. 2023-04-25 02:55:47,932 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:55:47,932 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:55:47,933 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:47,933 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:55:47,933 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:55:47,934 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:47,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:47,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:47,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:47,934 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:47,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:47,935 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:47,935 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:55:47,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:47,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:47,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:47,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:47,936 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:55:47,936 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@389d9736[Count = 0] remaining members to acquire global barrier 2023-04-25 02:55:47,936 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:55:47,936 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:47,938 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:47,938 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:47,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:47,938 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:55:47,938 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:55:47,938 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:55:47,940 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:55:47,940 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:47,940 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:47,940 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:55:47,941 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:55:47,941 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:55:47,941 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:55:47,941 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:55:47,941 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:55:47,941 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:55:47,941 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:55:47,942 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:47,942 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:47,942 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:47,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:47,943 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:47,943 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:47,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:47,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:47,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:47,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:47,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:47,944 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:47,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:47,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:47,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:47,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:47,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:47,945 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:47,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:47,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:47,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:47,948 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:47,948 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:47,948 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:47,948 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:47,948 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:47,948 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:47,948 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:47,948 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:47,948 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:47,948 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:47,948 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:55:47,948 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:55:47,949 DEBUG [Thread-2900] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:55:47,949 DEBUG [Thread-2900] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:55:47,952 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:47,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:47,953 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:47,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:48,046 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:48,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59162 deadline: 1682391408041, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:48,049 DEBUG [Thread-2900] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:55:48,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:55:48,051 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:55:48,052 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=233 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp/A/d117a9bf746a4d8595e45226929b8ab5 2023-04-25 02:55:48,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:55:48,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:55:48,052 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:55:48,052 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:48,052 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:55:48,052 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:55:48,053 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:48,053 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:48,054 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:48,054 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:48,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:48,054 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:48,055 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:48,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:48,055 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:55:48,055 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:48,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:55:48,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:48,056 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:55:48,056 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:55:48,056 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:55:48,056 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:55:48,056 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:48,056 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:55:48,058 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:55:48,058 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:55:48,058 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. 2023-04-25 02:55:48,058 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. started... 2023-04-25 02:55:48,058 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. as already flushing 2023-04-25 02:55:48,058 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. 2023-04-25 02:55:48,058 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:55:48,058 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:55:48,058 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:48,060 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:55:48,060 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:55:48,064 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:48,064 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:48,064 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:55:48,064 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:48,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:48,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:48,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:48,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:48,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:48,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:48,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:48,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:48,066 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:55:48,066 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5fdcfbe[Count = 0] remaining members to acquire global barrier 2023-04-25 02:55:48,066 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:55:48,066 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:48,067 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:48,067 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:48,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:48,067 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:55:48,067 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:55:48,067 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:55:48,067 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:48,067 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:55:48,069 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:55:48,069 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:48,069 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:55:48,069 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:48,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:48,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:48,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:48,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:48,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:48,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:48,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:48,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:48,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:48,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:48,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:55:48,073 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:55:48,073 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:55:48,073 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:55:48,073 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:55:48,073 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:55:48,073 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:55:48,078 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:48,078 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:48,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:48,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:48,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:48,078 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:48,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:48,078 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:48,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:48,078 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:48,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:48,079 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:48,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:48,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:48,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:48,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:48,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:48,080 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:48,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:48,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:48,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:48,085 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:48,085 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:48,085 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:48,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:48,086 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:48,086 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:48,086 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:48,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:48,086 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:48,086 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:55:48,086 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:55:48,086 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:48,086 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:48,086 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:48,086 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:48,086 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:48,086 DEBUG [Thread-2900] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:55:48,087 DEBUG [Thread-2900] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:55:48,187 DEBUG [Thread-2900] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:55:48,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:55:48,189 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:55:48,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:55:48,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:55:48,190 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:55:48,190 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:48,190 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:55:48,190 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:55:48,192 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:48,192 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:48,193 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:48,193 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:48,193 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:48,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:48,194 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:48,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:48,194 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:55:48,194 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:48,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:55:48,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:48,195 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:55:48,195 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:55:48,195 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:55:48,195 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:55:48,195 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:48,195 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:55:48,195 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:55:48,195 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:55:48,195 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. 2023-04-25 02:55:48,195 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. started... 2023-04-25 02:55:48,195 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. as already flushing 2023-04-25 02:55:48,195 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. 2023-04-25 02:55:48,196 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:55:48,196 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:55:48,196 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:48,196 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:55:48,196 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:55:48,197 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:48,198 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:48,198 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:55:48,198 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:48,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:48,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:48,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:48,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:48,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:48,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:48,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:48,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:48,201 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:55:48,201 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@c37ed4b[Count = 0] remaining members to acquire global barrier 2023-04-25 02:55:48,201 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:55:48,201 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:48,203 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:48,203 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:48,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:48,203 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:55:48,203 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:55:48,203 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:55:48,203 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:48,204 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:55:48,205 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:48,205 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:55:48,205 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:48,205 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:55:48,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:48,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:48,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:48,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:48,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:48,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:48,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:48,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:48,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:48,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:48,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:55:48,209 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:55:48,209 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:55:48,209 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:55:48,209 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:55:48,209 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:55:48,209 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:55:48,210 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:48,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:48,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:48,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:48,210 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:48,211 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:48,211 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:48,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:48,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:48,211 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:48,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:48,211 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:48,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:48,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:48,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:48,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:48,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:48,212 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:48,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:48,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:48,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:48,218 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:48,218 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:48,218 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:48,218 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:48,218 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:48,218 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:48,218 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:48,218 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:48,218 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:48,218 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:48,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:48,218 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:48,218 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:55:48,218 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:55:48,219 DEBUG [Thread-2900] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:55:48,219 DEBUG [Thread-2900] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:55:48,219 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:48,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:48,319 DEBUG [Thread-2900] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:55:48,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:55:48,328 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:55:48,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:55:48,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:55:48,330 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:55:48,330 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:48,330 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:55:48,330 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:55:48,331 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:48,331 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:48,334 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:48,334 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:48,334 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:48,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:48,334 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:48,334 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:55:48,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:48,335 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:48,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:55:48,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:48,335 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:55:48,335 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:55:48,335 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:55:48,335 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:55:48,336 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:48,336 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:55:48,345 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:55:48,345 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:55:48,345 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. 2023-04-25 02:55:48,345 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. started... 2023-04-25 02:55:48,345 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. as already flushing 2023-04-25 02:55:48,345 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. 2023-04-25 02:55:48,345 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:55:48,345 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:55:48,345 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:48,346 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:55:48,346 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:55:48,347 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:48,347 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:48,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:48,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:48,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:48,347 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:48,347 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:55:48,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:48,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:48,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:48,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:48,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:48,350 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:55:48,350 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7498b7e5[Count = 0] remaining members to acquire global barrier 2023-04-25 02:55:48,350 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:55:48,350 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:48,352 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:48,352 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:48,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:48,352 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:55:48,352 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:48,352 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:55:48,352 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:55:48,352 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:55:48,354 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:55:48,354 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:48,354 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:55:48,354 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:48,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:48,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:48,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:48,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:48,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:48,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:48,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:48,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:48,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:48,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:48,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:55:48,356 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:55:48,357 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:55:48,357 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:55:48,357 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:55:48,357 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:55:48,357 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:55:48,357 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:48,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59162 deadline: 1682391408352, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:48,361 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:48,361 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:48,361 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:48,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:48,361 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:48,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:48,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:48,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:48,361 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:48,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:48,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:48,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:48,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:48,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:48,363 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:48,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:48,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:48,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:48,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:48,365 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:48,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:48,367 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:48,367 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:48,367 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:48,367 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:48,368 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:48,368 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:48,368 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:48,368 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:55:48,368 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:55:48,368 DEBUG [Thread-2900] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:55:48,368 DEBUG [Thread-2900] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:55:48,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:48,368 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:48,368 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:48,368 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:48,368 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:48,369 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:48,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:48,468 DEBUG [Thread-2900] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:55:48,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:55:48,470 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:55:48,472 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:55:48,472 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:55:48,472 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:55:48,472 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:48,472 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:55:48,472 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:55:48,473 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:48,473 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:48,474 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:48,474 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:48,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:48,474 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:48,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:48,475 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:48,475 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:55:48,475 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:48,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:55:48,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:48,476 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:55:48,480 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:55:48,480 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:55:48,480 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:55:48,480 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:48,480 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:55:48,484 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:55:48,484 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:55:48,484 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. 2023-04-25 02:55:48,484 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. started... 2023-04-25 02:55:48,484 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. as already flushing 2023-04-25 02:55:48,484 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. 2023-04-25 02:55:48,484 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:55:48,484 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:55:48,484 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:48,484 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:55:48,484 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:55:48,485 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=233 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp/B/447acc876db94cd584af1700e4ad7539 2023-04-25 02:55:48,487 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:48,487 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:48,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:48,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:48,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:48,488 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:48,488 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:55:48,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:48,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:48,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:48,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:48,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:48,489 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:55:48,489 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@20e8ba67[Count = 0] remaining members to acquire global barrier 2023-04-25 02:55:48,489 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:55:48,489 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:48,492 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:48,492 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:48,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:48,492 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:55:48,492 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:55:48,492 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:55:48,494 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:48,494 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:55:48,495 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:48,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:48,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:48,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:48,495 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:55:48,495 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:48,495 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:55:48,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:48,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:48,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:48,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:48,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:48,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:48,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:48,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:55:48,499 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:55:48,499 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:55:48,499 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:55:48,499 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:55:48,499 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:55:48,499 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:55:48,506 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:48,506 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:48,506 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:48,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:48,506 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:48,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:48,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:48,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:48,506 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:48,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:48,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:48,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:48,507 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:48,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:48,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:48,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:48,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:48,509 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:48,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:48,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:48,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:48,518 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=233 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp/C/664d7144148f4c78b163589760ecbb63 2023-04-25 02:55:48,521 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:48,521 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:48,521 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:48,521 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:48,521 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:48,521 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:48,521 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:48,521 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:48,521 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:48,521 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:48,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:48,521 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:55:48,522 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:55:48,522 DEBUG [Thread-2900] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:55:48,522 DEBUG [Thread-2900] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:55:48,521 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:48,522 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:48,522 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:55:48,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:48,524 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp/A/d117a9bf746a4d8595e45226929b8ab5 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/d117a9bf746a4d8595e45226929b8ab5 2023-04-25 02:55:48,530 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/d117a9bf746a4d8595e45226929b8ab5, entries=200, sequenceid=233, filesize=14.1 K 2023-04-25 02:55:48,531 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp/B/447acc876db94cd584af1700e4ad7539 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/447acc876db94cd584af1700e4ad7539 2023-04-25 02:55:48,535 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/447acc876db94cd584af1700e4ad7539, entries=150, sequenceid=233, filesize=11.8 K 2023-04-25 02:55:48,536 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp/C/664d7144148f4c78b163589760ecbb63 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/664d7144148f4c78b163589760ecbb63 2023-04-25 02:55:48,539 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/664d7144148f4c78b163589760ecbb63, entries=150, sequenceid=233, filesize=11.8 K 2023-04-25 02:55:48,540 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for 5f2c97e00c770d329321cff2fbd6848c in 901ms, sequenceid=233, compaction requested=true 2023-04-25 02:55:48,540 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 5f2c97e00c770d329321cff2fbd6848c: 2023-04-25 02:55:48,540 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 02:55:48,540 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-25 02:55:48,540 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 02:55:48,540 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-25 02:55:48,540 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 02:55:48,541 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 38840 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-25 02:55:48,541 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36450 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-25 02:55:48,541 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HStore(1912): 5f2c97e00c770d329321cff2fbd6848c/A is initiating minor compaction (all files) 2023-04-25 02:55:48,541 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HStore(1912): 5f2c97e00c770d329321cff2fbd6848c/B is initiating minor compaction (all files) 2023-04-25 02:55:48,541 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 5f2c97e00c770d329321cff2fbd6848c/A in TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. 2023-04-25 02:55:48,541 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 5f2c97e00c770d329321cff2fbd6848c/B in TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. 2023-04-25 02:55:48,541 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/52e807f0534b46299f8e5373f67266ba, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/287d09e1d924469b9e6a75a83fb1dd38, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/d117a9bf746a4d8595e45226929b8ab5] into tmpdir=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp, totalSize=37.9 K 2023-04-25 02:55:48,541 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/932e44e415b14d86aced9cb57cabdef5, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/dbb2305ba9404b9aa60db115f822326c, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/447acc876db94cd584af1700e4ad7539] into tmpdir=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp, totalSize=35.6 K 2023-04-25 02:55:48,542 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting 52e807f0534b46299f8e5373f67266ba, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=196, earliestPutTs=1682391345310 2023-04-25 02:55:48,542 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting 932e44e415b14d86aced9cb57cabdef5, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=196, earliestPutTs=1682391345310 2023-04-25 02:55:48,542 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting 287d09e1d924469b9e6a75a83fb1dd38, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=217, earliestPutTs=1682391345569 2023-04-25 02:55:48,542 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting dbb2305ba9404b9aa60db115f822326c, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=217, earliestPutTs=1682391345569 2023-04-25 02:55:48,542 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting d117a9bf746a4d8595e45226929b8ab5, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=233, earliestPutTs=1682391346752 2023-04-25 02:55:48,543 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting 447acc876db94cd584af1700e4ad7539, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=233, earliestPutTs=1682391346752 2023-04-25 02:55:48,553 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] throttle.PressureAwareThroughputController(145): 5f2c97e00c770d329321cff2fbd6848c#B#compaction#414 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 53.85 MB/second 2023-04-25 02:55:48,553 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] throttle.PressureAwareThroughputController(145): 5f2c97e00c770d329321cff2fbd6848c#A#compaction#415 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-25 02:55:48,622 DEBUG [Thread-2900] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:55:48,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:55:48,625 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:55:48,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:55:48,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:55:48,626 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:55:48,626 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:48,626 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:55:48,626 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:55:48,626 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:48,627 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:48,629 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:48,629 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:48,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:48,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:48,629 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:48,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:55:48,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:48,630 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:55:48,630 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:48,630 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:55:48,630 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:55:48,630 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:55:48,630 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:48,630 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:55:48,631 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:55:48,631 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:55:48,631 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:48,631 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:55:48,632 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. 2023-04-25 02:55:48,632 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. started... 2023-04-25 02:55:48,632 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5f2c97e00c770d329321cff2fbd6848c 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-25 02:55:48,632 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5f2c97e00c770d329321cff2fbd6848c, store=A 2023-04-25 02:55:48,632 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:55:48,632 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5f2c97e00c770d329321cff2fbd6848c, store=B 2023-04-25 02:55:48,633 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:55:48,633 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5f2c97e00c770d329321cff2fbd6848c, store=C 2023-04-25 02:55:48,633 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:55:48,648 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=250 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp/A/d8598b45d90641cba7565004cbc1935d 2023-04-25 02:55:48,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(9158): Flush requested on 5f2c97e00c770d329321cff2fbd6848c 2023-04-25 02:55:48,866 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. as already flushing 2023-04-25 02:55:48,934 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:48,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59114 deadline: 1682391408923, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:48,934 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:48,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59162 deadline: 1682391408923, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:48,935 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:48,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59156 deadline: 1682391408925, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:48,935 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:48,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59136 deadline: 1682391408927, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:48,940 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:48,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59116 deadline: 1682391408934, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:48,983 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp/B/c445318e69234d5887febfa5e5631384 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/c445318e69234d5887febfa5e5631384 2023-04-25 02:55:48,985 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp/A/4fa064df1cdb48d48fc39f053e8f8992 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/4fa064df1cdb48d48fc39f053e8f8992 2023-04-25 02:55:48,988 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 5f2c97e00c770d329321cff2fbd6848c/B of 5f2c97e00c770d329321cff2fbd6848c into c445318e69234d5887febfa5e5631384(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 02:55:48,988 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5f2c97e00c770d329321cff2fbd6848c: 2023-04-25 02:55:48,988 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c., storeName=5f2c97e00c770d329321cff2fbd6848c/B, priority=13, startTime=1682391348540; duration=0sec 2023-04-25 02:55:48,989 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 02:55:48,989 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-25 02:55:48,989 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36450 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-25 02:55:48,990 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HStore(1912): 5f2c97e00c770d329321cff2fbd6848c/C is initiating minor compaction (all files) 2023-04-25 02:55:48,990 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 5f2c97e00c770d329321cff2fbd6848c/C in TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. 2023-04-25 02:55:48,990 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/dc77376dad9d4835bdf1c9fa7fa07407, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/d53625636187492a9e6e3f4adabf357a, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/664d7144148f4c78b163589760ecbb63] into tmpdir=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp, totalSize=35.6 K 2023-04-25 02:55:48,990 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting dc77376dad9d4835bdf1c9fa7fa07407, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=196, earliestPutTs=1682391345310 2023-04-25 02:55:48,991 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting d53625636187492a9e6e3f4adabf357a, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=217, earliestPutTs=1682391345569 2023-04-25 02:55:48,991 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 5f2c97e00c770d329321cff2fbd6848c/A of 5f2c97e00c770d329321cff2fbd6848c into 4fa064df1cdb48d48fc39f053e8f8992(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 02:55:48,991 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5f2c97e00c770d329321cff2fbd6848c: 2023-04-25 02:55:48,991 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c., storeName=5f2c97e00c770d329321cff2fbd6848c/A, priority=13, startTime=1682391348540; duration=0sec 2023-04-25 02:55:48,991 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 02:55:48,991 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting 664d7144148f4c78b163589760ecbb63, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=233, earliestPutTs=1682391346752 2023-04-25 02:55:49,003 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] throttle.PressureAwareThroughputController(145): 5f2c97e00c770d329321cff2fbd6848c#C#compaction#418 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-25 02:55:49,018 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp/C/3379d58dc5514ea2a5beb85cd4a13556 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/3379d58dc5514ea2a5beb85cd4a13556 2023-04-25 02:55:49,023 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 5f2c97e00c770d329321cff2fbd6848c/C of 5f2c97e00c770d329321cff2fbd6848c into 3379d58dc5514ea2a5beb85cd4a13556(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 02:55:49,023 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5f2c97e00c770d329321cff2fbd6848c: 2023-04-25 02:55:49,023 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c., storeName=5f2c97e00c770d329321cff2fbd6848c/C, priority=13, startTime=1682391348540; duration=0sec 2023-04-25 02:55:49,024 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 02:55:49,039 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:49,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59162 deadline: 1682391409036, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:49,040 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:49,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59136 deadline: 1682391409036, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:49,045 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:49,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59116 deadline: 1682391409042, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:49,069 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=250 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp/B/4218a8678fdf4ec2963e4bf68489839b 2023-04-25 02:55:49,098 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=250 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp/C/a00dc3705484472b9a6e4608a6b2c4f4 2023-04-25 02:55:49,105 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp/A/d8598b45d90641cba7565004cbc1935d as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/d8598b45d90641cba7565004cbc1935d 2023-04-25 02:55:49,109 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/d8598b45d90641cba7565004cbc1935d, entries=150, sequenceid=250, filesize=11.8 K 2023-04-25 02:55:49,109 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp/B/4218a8678fdf4ec2963e4bf68489839b as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/4218a8678fdf4ec2963e4bf68489839b 2023-04-25 02:55:49,116 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/4218a8678fdf4ec2963e4bf68489839b, entries=150, sequenceid=250, filesize=11.8 K 2023-04-25 02:55:49,116 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.StoreScanner(961): StoreScanner already closing. There is no need to updateReaders 2023-04-25 02:55:49,117 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp/C/a00dc3705484472b9a6e4608a6b2c4f4 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/a00dc3705484472b9a6e4608a6b2c4f4 2023-04-25 02:55:49,120 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/a00dc3705484472b9a6e4608a6b2c4f4, entries=150, sequenceid=250, filesize=11.8 K 2023-04-25 02:55:49,120 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for 5f2c97e00c770d329321cff2fbd6848c in 488ms, sequenceid=250, compaction requested=false 2023-04-25 02:55:49,120 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5f2c97e00c770d329321cff2fbd6848c: 2023-04-25 02:55:49,120 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. 2023-04-25 02:55:49,120 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:55:49,120 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:55:49,120 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:49,120 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:55:49,120 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:55:49,123 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:49,123 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:49,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:49,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:49,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:49,123 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:49,123 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:55:49,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:49,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:49,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:49,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:49,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:49,125 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:55:49,125 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4b0301fb[Count = 0] remaining members to acquire global barrier 2023-04-25 02:55:49,125 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:55:49,125 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:49,126 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:49,126 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:49,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:49,126 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:55:49,126 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:55:49,126 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:55:49,127 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:49,127 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:55:49,133 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:49,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:49,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:49,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:49,133 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:55:49,133 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:49,133 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:55:49,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:49,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:49,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:49,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:49,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:49,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:49,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:49,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:55:49,137 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:55:49,137 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:55:49,137 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:55:49,137 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:55:49,137 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:55:49,137 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:55:49,138 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:49,138 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:49,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:49,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:49,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:49,138 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:49,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:49,138 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:49,139 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:49,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:49,139 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:49,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:49,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:49,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:49,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:49,140 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:49,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:49,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:49,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:49,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:49,144 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:49,144 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:49,144 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:49,144 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:49,144 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:49,144 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:49,144 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:49,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:49,144 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:49,144 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:49,144 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:55:49,144 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:55:49,144 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:49,144 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:49,144 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:49,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:49,144 DEBUG [Thread-2900] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:55:49,144 DEBUG [Thread-2900] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:55:49,144 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:55:49,244 DEBUG [Thread-2900] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:55:49,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:55:49,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(9158): Flush requested on 5f2c97e00c770d329321cff2fbd6848c 2023-04-25 02:55:49,246 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 5f2c97e00c770d329321cff2fbd6848c 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-25 02:55:49,246 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5f2c97e00c770d329321cff2fbd6848c, store=A 2023-04-25 02:55:49,246 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:55:49,246 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5f2c97e00c770d329321cff2fbd6848c, store=B 2023-04-25 02:55:49,246 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:55:49,246 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5f2c97e00c770d329321cff2fbd6848c, store=C 2023-04-25 02:55:49,246 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:55:49,247 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:55:49,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:55:49,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:55:49,248 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:55:49,248 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:49,248 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:55:49,248 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:55:49,249 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:49,250 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:49,251 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:49,251 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:49,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:49,251 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:49,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:49,252 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:49,252 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:55:49,252 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:49,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:55:49,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:49,253 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:55:49,253 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:55:49,253 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:55:49,253 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:55:49,253 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:49,253 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:55:49,256 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:55:49,256 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:55:49,256 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. 2023-04-25 02:55:49,256 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. started... 2023-04-25 02:55:49,256 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. as already flushing 2023-04-25 02:55:49,256 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. 2023-04-25 02:55:49,256 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:55:49,256 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:55:49,256 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:49,264 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:55:49,264 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:55:49,266 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:49,266 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:49,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:49,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:49,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:49,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:49,267 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:49,267 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:55:49,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:49,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:49,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:49,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:49,269 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:55:49,269 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@786de065[Count = 0] remaining members to acquire global barrier 2023-04-25 02:55:49,269 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:55:49,269 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:49,270 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:49,270 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:49,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:49,270 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:55:49,270 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:55:49,270 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:55:49,270 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:49,270 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:55:49,273 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:49,273 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:55:49,273 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:49,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:49,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:49,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:49,273 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:55:49,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:49,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:49,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:49,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:49,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:49,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:49,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:49,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:55:49,276 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:55:49,276 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:55:49,276 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:55:49,276 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:55:49,276 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:55:49,276 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:55:49,277 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:49,277 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:49,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:49,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:49,277 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:49,277 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:49,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:49,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:49,278 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:49,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:49,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:49,278 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:49,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:49,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:49,282 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:49,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59162 deadline: 1682391409272, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:49,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:49,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:49,285 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:49,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:49,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:49,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:49,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:49,288 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:49,288 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:49,288 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:49,288 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:49,288 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:49,288 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:49,288 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:49,288 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:49,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:49,288 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:49,288 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:49,288 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:49,289 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:55:49,289 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:55:49,289 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:49,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:49,289 DEBUG [Thread-2900] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:55:49,289 DEBUG [Thread-2900] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:55:49,289 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:55:49,291 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:49,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59116 deadline: 1682391409282, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:49,291 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:49,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59136 deadline: 1682391409284, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:49,389 DEBUG [Thread-2900] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:55:49,390 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:49,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:55:49,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59162 deadline: 1682391409383, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:49,391 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:55:49,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:55:49,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:55:49,392 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:55:49,392 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:49,392 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:55:49,392 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:55:49,393 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:49,393 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:49,394 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:49,394 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:49,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:49,394 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:49,395 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:49,395 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:55:49,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:49,395 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:49,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:55:49,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:49,395 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:55:49,396 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:55:49,396 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:55:49,396 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:55:49,396 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:49,396 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:55:49,396 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:55:49,396 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:55:49,396 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. 2023-04-25 02:55:49,396 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. started... 2023-04-25 02:55:49,396 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. as already flushing 2023-04-25 02:55:49,396 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. 2023-04-25 02:55:49,396 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:55:49,396 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:55:49,396 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:49,396 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:55:49,396 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:55:49,398 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:49,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:49,398 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:49,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:49,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:49,398 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:49,398 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:55:49,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:49,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:49,399 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:49,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59116 deadline: 1682391409393, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:49,399 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:49,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59136 deadline: 1682391409393, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:49,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:49,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:49,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:49,401 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:55:49,401 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@49fb5712[Count = 0] remaining members to acquire global barrier 2023-04-25 02:55:49,401 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:55:49,401 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:49,402 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:49,402 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:49,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:49,402 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:55:49,402 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:55:49,402 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:55:49,402 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:49,402 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:55:49,405 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:55:49,405 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:49,405 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:49,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:49,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:49,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:49,405 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:55:49,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:49,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:49,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:49,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:49,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:49,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:49,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:49,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:55:49,407 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:55:49,407 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:55:49,407 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:55:49,407 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:55:49,407 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:55:49,407 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:55:49,410 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:49,410 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:49,410 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:49,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:49,410 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:49,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:49,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:49,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:49,411 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:49,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:49,411 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:49,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:49,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:49,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:49,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:49,412 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:49,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:49,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:49,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:49,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:49,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:49,417 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:49,417 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:49,417 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:49,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:49,417 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:55:49,417 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:49,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:49,418 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:49,418 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:49,418 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:49,418 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:49,418 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:49,418 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:49,418 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:49,418 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:49,418 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:55:49,418 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:55:49,419 DEBUG [Thread-2900] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:55:49,419 DEBUG [Thread-2900] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:55:49,519 DEBUG [Thread-2900] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:55:49,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:55:49,521 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:55:49,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:55:49,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:55:49,522 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:55:49,522 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:49,522 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:55:49,522 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:55:49,522 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:49,522 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:49,524 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:49,524 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:49,524 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:49,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:49,525 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:49,525 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:55:49,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:49,525 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:49,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:55:49,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:49,526 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:55:49,526 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:55:49,526 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:55:49,526 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:55:49,526 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:49,526 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:55:49,526 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:55:49,526 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:55:49,526 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. 2023-04-25 02:55:49,526 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. started... 2023-04-25 02:55:49,526 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. as already flushing 2023-04-25 02:55:49,526 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. 2023-04-25 02:55:49,526 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:55:49,526 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:55:49,526 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:49,527 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:55:49,527 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:55:49,528 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:49,528 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:49,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:49,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:49,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:49,528 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:49,528 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:55:49,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:49,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:49,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:49,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:49,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:49,531 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:55:49,531 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2ca5f7c8[Count = 0] remaining members to acquire global barrier 2023-04-25 02:55:49,531 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:55:49,531 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:49,532 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:49,532 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:49,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:49,532 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:55:49,532 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:55:49,532 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:55:49,533 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:49,533 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:55:49,534 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:49,534 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:55:49,534 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:49,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:49,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:49,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:49,534 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:55:49,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:49,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:49,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:49,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:49,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:49,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:49,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:49,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:55:49,536 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:55:49,536 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:55:49,536 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:55:49,536 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:55:49,536 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:55:49,536 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:55:49,538 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:49,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:49,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:49,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:49,538 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:49,538 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:49,538 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:49,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:49,539 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:49,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:49,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:49,539 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:49,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:49,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:49,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:49,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:49,540 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:49,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:49,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:49,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:49,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:49,542 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:49,542 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:49,542 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:49,542 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:49,542 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:49,542 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:49,543 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:49,543 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:55:49,544 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:49,547 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:55:49,547 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:55:49,547 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:49,547 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:49,547 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:49,547 DEBUG [Thread-2900] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:55:49,547 DEBUG [Thread-2900] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:55:49,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:49,548 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:49,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:49,596 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:49,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59162 deadline: 1682391409592, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:49,603 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:49,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59116 deadline: 1682391409600, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:49,603 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:49,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59136 deadline: 1682391409600, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:49,648 DEBUG [Thread-2900] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:55:49,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:55:49,650 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:55:49,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:55:49,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:55:49,651 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:55:49,651 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:49,651 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:55:49,651 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:55:49,652 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:49,652 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:49,653 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:49,653 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:49,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:49,653 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:49,653 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:49,653 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:55:49,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:49,654 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:49,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:55:49,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:49,654 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:55:49,654 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:55:49,654 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:55:49,654 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:55:49,655 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:49,655 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:55:49,655 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:55:49,655 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:55:49,656 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. 2023-04-25 02:55:49,656 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. started... 2023-04-25 02:55:49,656 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. as already flushing 2023-04-25 02:55:49,656 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. 2023-04-25 02:55:49,656 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:55:49,656 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:55:49,656 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:49,656 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:55:49,656 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:55:49,658 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:49,658 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:49,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:49,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:49,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:49,658 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:49,658 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:55:49,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:49,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:49,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:49,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:49,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:49,659 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:55:49,659 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7ffa377e[Count = 0] remaining members to acquire global barrier 2023-04-25 02:55:49,660 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:55:49,660 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:49,663 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:49,663 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:49,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:49,663 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:55:49,663 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:55:49,663 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:55:49,663 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:49,663 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:55:49,664 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:55:49,664 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:49,664 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:49,664 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:55:49,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:49,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:49,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:49,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:49,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:49,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:49,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:49,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:49,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:49,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:49,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:55:49,666 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:55:49,666 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:55:49,666 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:55:49,666 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:55:49,666 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:55:49,667 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:55:49,669 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=270 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp/A/2dd6823cb0444575b92b8c48203402dc 2023-04-25 02:55:49,672 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:49,672 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:49,673 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:49,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:49,673 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:49,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:49,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:49,674 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:49,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:49,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:49,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:49,675 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:49,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:49,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:49,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:49,678 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:49,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:49,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:49,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:49,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:49,683 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:49,683 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:49,683 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:49,684 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:49,684 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:49,684 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:49,684 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:49,684 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:55:49,684 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:55:49,684 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:49,684 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:49,684 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:49,684 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:49,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:49,684 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:49,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:49,684 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:55:49,684 DEBUG [Thread-2900] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:55:49,686 DEBUG [Thread-2900] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:55:49,694 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=270 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp/B/d26188e865fa4301becb61a3c08233a3 2023-04-25 02:55:49,714 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=270 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp/C/696a9b716ea84776b15e96c7a7f9b67e 2023-04-25 02:55:49,720 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp/A/2dd6823cb0444575b92b8c48203402dc as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/2dd6823cb0444575b92b8c48203402dc 2023-04-25 02:55:49,724 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/2dd6823cb0444575b92b8c48203402dc, entries=200, sequenceid=270, filesize=14.3 K 2023-04-25 02:55:49,725 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp/B/d26188e865fa4301becb61a3c08233a3 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/d26188e865fa4301becb61a3c08233a3 2023-04-25 02:55:49,729 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/d26188e865fa4301becb61a3c08233a3, entries=100, sequenceid=270, filesize=9.5 K 2023-04-25 02:55:49,730 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp/C/696a9b716ea84776b15e96c7a7f9b67e as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/696a9b716ea84776b15e96c7a7f9b67e 2023-04-25 02:55:49,737 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/696a9b716ea84776b15e96c7a7f9b67e, entries=100, sequenceid=270, filesize=9.5 K 2023-04-25 02:55:49,738 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=87.22 KB/89310 for 5f2c97e00c770d329321cff2fbd6848c in 492ms, sequenceid=270, compaction requested=true 2023-04-25 02:55:49,738 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 5f2c97e00c770d329321cff2fbd6848c: 2023-04-25 02:55:49,738 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 02:55:49,738 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-25 02:55:49,738 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-25 02:55:49,738 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 02:55:49,739 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 02:55:49,739 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39138 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-25 02:55:49,739 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34258 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-25 02:55:49,740 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HStore(1912): 5f2c97e00c770d329321cff2fbd6848c/B is initiating minor compaction (all files) 2023-04-25 02:55:49,740 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 5f2c97e00c770d329321cff2fbd6848c/B in TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. 2023-04-25 02:55:49,740 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/c445318e69234d5887febfa5e5631384, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/4218a8678fdf4ec2963e4bf68489839b, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/d26188e865fa4301becb61a3c08233a3] into tmpdir=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp, totalSize=33.5 K 2023-04-25 02:55:49,739 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HStore(1912): 5f2c97e00c770d329321cff2fbd6848c/A is initiating minor compaction (all files) 2023-04-25 02:55:49,740 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 5f2c97e00c770d329321cff2fbd6848c/A in TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. 2023-04-25 02:55:49,740 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/4fa064df1cdb48d48fc39f053e8f8992, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/d8598b45d90641cba7565004cbc1935d, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/2dd6823cb0444575b92b8c48203402dc] into tmpdir=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp, totalSize=38.2 K 2023-04-25 02:55:49,740 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting c445318e69234d5887febfa5e5631384, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=233, earliestPutTs=1682391346752 2023-04-25 02:55:49,741 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting 4fa064df1cdb48d48fc39f053e8f8992, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=233, earliestPutTs=1682391346752 2023-04-25 02:55:49,741 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting 4218a8678fdf4ec2963e4bf68489839b, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=250, earliestPutTs=1682391347670 2023-04-25 02:55:49,741 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting d8598b45d90641cba7565004cbc1935d, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=250, earliestPutTs=1682391347670 2023-04-25 02:55:49,741 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting d26188e865fa4301becb61a3c08233a3, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=270, earliestPutTs=1682391348921 2023-04-25 02:55:49,742 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting 2dd6823cb0444575b92b8c48203402dc, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=270, earliestPutTs=1682391348921 2023-04-25 02:55:49,748 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] throttle.PressureAwareThroughputController(145): 5f2c97e00c770d329321cff2fbd6848c#B#compaction#423 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-25 02:55:49,749 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] throttle.PressureAwareThroughputController(145): 5f2c97e00c770d329321cff2fbd6848c#A#compaction#424 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-25 02:55:49,778 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp/B/56d61f70a2d4416ab32bef05bd696a21 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/56d61f70a2d4416ab32bef05bd696a21 2023-04-25 02:55:49,782 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 5f2c97e00c770d329321cff2fbd6848c/B of 5f2c97e00c770d329321cff2fbd6848c into 56d61f70a2d4416ab32bef05bd696a21(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 02:55:49,782 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5f2c97e00c770d329321cff2fbd6848c: 2023-04-25 02:55:49,782 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c., storeName=5f2c97e00c770d329321cff2fbd6848c/B, priority=13, startTime=1682391349738; duration=0sec 2023-04-25 02:55:49,782 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 02:55:49,782 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-25 02:55:49,784 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34258 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-25 02:55:49,785 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HStore(1912): 5f2c97e00c770d329321cff2fbd6848c/C is initiating minor compaction (all files) 2023-04-25 02:55:49,785 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 5f2c97e00c770d329321cff2fbd6848c/C in TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. 2023-04-25 02:55:49,785 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/3379d58dc5514ea2a5beb85cd4a13556, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/a00dc3705484472b9a6e4608a6b2c4f4, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/696a9b716ea84776b15e96c7a7f9b67e] into tmpdir=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp, totalSize=33.5 K 2023-04-25 02:55:49,785 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting 3379d58dc5514ea2a5beb85cd4a13556, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=233, earliestPutTs=1682391346752 2023-04-25 02:55:49,785 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting a00dc3705484472b9a6e4608a6b2c4f4, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=250, earliestPutTs=1682391347670 2023-04-25 02:55:49,786 DEBUG [Thread-2900] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:55:49,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:55:49,787 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting 696a9b716ea84776b15e96c7a7f9b67e, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=270, earliestPutTs=1682391348921 2023-04-25 02:55:49,788 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:55:49,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:55:49,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:55:49,789 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:55:49,789 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:49,789 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:55:49,789 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:55:49,790 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:49,790 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:49,792 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:49,792 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:49,792 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:49,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:49,793 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:49,793 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:55:49,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:49,793 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:49,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:55:49,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:49,793 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:55:49,797 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] throttle.PressureAwareThroughputController(145): 5f2c97e00c770d329321cff2fbd6848c#C#compaction#425 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-25 02:55:49,804 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:55:49,804 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:55:49,804 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:55:49,804 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:49,804 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:55:49,808 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:55:49,808 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:55:49,808 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. 2023-04-25 02:55:49,808 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. started... 2023-04-25 02:55:49,808 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5f2c97e00c770d329321cff2fbd6848c 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-25 02:55:49,808 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5f2c97e00c770d329321cff2fbd6848c, store=A 2023-04-25 02:55:49,808 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:55:49,808 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5f2c97e00c770d329321cff2fbd6848c, store=B 2023-04-25 02:55:49,809 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:55:49,809 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5f2c97e00c770d329321cff2fbd6848c, store=C 2023-04-25 02:55:49,809 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:55:49,833 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp/C/7f506986abab4b5089b3c8f0ddcc4400 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/7f506986abab4b5089b3c8f0ddcc4400 2023-04-25 02:55:49,839 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 5f2c97e00c770d329321cff2fbd6848c/C of 5f2c97e00c770d329321cff2fbd6848c into 7f506986abab4b5089b3c8f0ddcc4400(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 02:55:49,839 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5f2c97e00c770d329321cff2fbd6848c: 2023-04-25 02:55:49,839 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c., storeName=5f2c97e00c770d329321cff2fbd6848c/C, priority=13, startTime=1682391349739; duration=0sec 2023-04-25 02:55:49,839 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 02:55:49,856 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=287 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp/A/29dc053dd8cf4baca8fb1bcc48973535 2023-04-25 02:55:49,885 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=287 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp/B/9a6aa4970f514d45a22dbd79b8707a39 2023-04-25 02:55:49,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(9158): Flush requested on 5f2c97e00c770d329321cff2fbd6848c 2023-04-25 02:55:49,903 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. as already flushing 2023-04-25 02:55:49,905 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=287 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp/C/c27b7be3b60c421ca54880277c0f73ce 2023-04-25 02:55:49,909 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp/A/29dc053dd8cf4baca8fb1bcc48973535 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/29dc053dd8cf4baca8fb1bcc48973535 2023-04-25 02:55:49,912 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/29dc053dd8cf4baca8fb1bcc48973535, entries=150, sequenceid=287, filesize=11.9 K 2023-04-25 02:55:49,913 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp/B/9a6aa4970f514d45a22dbd79b8707a39 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/9a6aa4970f514d45a22dbd79b8707a39 2023-04-25 02:55:49,919 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/9a6aa4970f514d45a22dbd79b8707a39, entries=150, sequenceid=287, filesize=11.9 K 2023-04-25 02:55:49,920 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp/C/c27b7be3b60c421ca54880277c0f73ce as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/c27b7be3b60c421ca54880277c0f73ce 2023-04-25 02:55:49,927 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/c27b7be3b60c421ca54880277c0f73ce, entries=150, sequenceid=287, filesize=11.9 K 2023-04-25 02:55:49,931 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=40.25 KB/41220 for 5f2c97e00c770d329321cff2fbd6848c in 123ms, sequenceid=287, compaction requested=false 2023-04-25 02:55:49,931 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5f2c97e00c770d329321cff2fbd6848c: 2023-04-25 02:55:49,931 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. 2023-04-25 02:55:49,932 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:55:49,932 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:55:49,932 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:49,932 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:55:49,932 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:55:49,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(9158): Flush requested on 5f2c97e00c770d329321cff2fbd6848c 2023-04-25 02:55:49,933 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 5f2c97e00c770d329321cff2fbd6848c 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-25 02:55:49,934 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:49,934 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:49,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:49,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:49,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:49,934 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:49,934 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:55:49,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:49,934 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5f2c97e00c770d329321cff2fbd6848c, store=A 2023-04-25 02:55:49,934 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:55:49,935 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5f2c97e00c770d329321cff2fbd6848c, store=B 2023-04-25 02:55:49,935 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:55:49,935 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5f2c97e00c770d329321cff2fbd6848c, store=C 2023-04-25 02:55:49,935 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:55:49,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:49,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:49,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:49,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:49,936 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:55:49,936 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@68838e2c[Count = 0] remaining members to acquire global barrier 2023-04-25 02:55:49,936 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:55:49,936 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:49,937 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:49,937 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:49,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:49,938 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:55:49,938 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:55:49,938 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:55:49,938 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:49,938 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:55:49,939 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:49,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:49,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:49,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:49,939 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:55:49,939 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:49,939 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:55:49,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:49,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:49,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:49,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:49,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:49,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:49,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:49,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:55:49,944 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:55:49,944 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:55:49,944 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:55:49,944 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:55:49,944 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:55:49,944 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:55:49,950 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:49,950 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:49,950 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:49,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:49,950 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:49,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:49,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:49,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:49,950 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:49,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:49,950 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:49,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:49,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:49,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:49,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:49,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:49,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:49,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:49,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:49,962 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:49,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:49,964 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:49,964 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:49,964 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:49,964 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:49,964 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:49,964 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:49,964 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:49,964 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:49,964 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:55:49,964 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:55:49,965 DEBUG [Thread-2900] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:55:49,965 DEBUG [Thread-2900] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:55:49,965 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:49,965 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:49,965 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:49,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:49,966 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:49,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:50,032 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:50,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59162 deadline: 1682391410024, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:50,034 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:50,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59116 deadline: 1682391410025, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:50,037 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:50,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59136 deadline: 1682391410032, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:50,065 DEBUG [Thread-2900] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:55:50,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:55:50,067 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:55:50,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:55:50,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:55:50,068 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:55:50,068 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:50,068 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:55:50,068 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:55:50,069 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:50,069 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:50,070 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:50,070 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:50,070 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:50,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:50,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:50,071 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:50,071 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:55:50,071 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:50,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:55:50,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:50,072 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:55:50,072 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:55:50,072 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:55:50,072 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:50,072 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:55:50,072 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:55:50,072 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:55:50,072 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. 2023-04-25 02:55:50,072 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. started... 2023-04-25 02:55:50,072 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. as already flushing 2023-04-25 02:55:50,072 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. 2023-04-25 02:55:50,073 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:55:50,073 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:55:50,073 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:50,073 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:55:50,073 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:55:50,078 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:50,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:50,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:50,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:50,079 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:50,079 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:50,079 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:55:50,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:50,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:50,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:50,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:50,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:50,081 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:55:50,081 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7bd317e9[Count = 0] remaining members to acquire global barrier 2023-04-25 02:55:50,081 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:55:50,082 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:50,085 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:50,085 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:50,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:50,085 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:55:50,085 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:55:50,085 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:55:50,087 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:50,087 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:55:50,087 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:50,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:50,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:50,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:50,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:50,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:50,089 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:55:50,089 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:50,089 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:55:50,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:50,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:50,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:50,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:50,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:50,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:55:50,092 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:55:50,092 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:55:50,092 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:55:50,092 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:55:50,092 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:55:50,092 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:55:50,094 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:50,094 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:50,094 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:50,094 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:50,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:50,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:50,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:50,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:50,094 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:50,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:50,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:50,095 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:50,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:50,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:50,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:50,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:50,097 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:50,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:50,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:50,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:50,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:50,101 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:50,102 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:50,102 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:50,102 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:55:50,102 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:50,102 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:50,102 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:55:50,102 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:55:50,102 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:50,102 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:50,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:50,102 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:50,102 DEBUG [Thread-2900] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:55:50,102 DEBUG [Thread-2900] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:55:50,102 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:50,102 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:50,102 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:50,102 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:50,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:50,138 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:50,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59162 deadline: 1682391410133, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:50,140 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:50,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59116 deadline: 1682391410135, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:50,145 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:50,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59136 deadline: 1682391410138, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:50,182 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp/A/e960ee71989d4b839dc250da7ef43fc9 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/e960ee71989d4b839dc250da7ef43fc9 2023-04-25 02:55:50,185 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 5f2c97e00c770d329321cff2fbd6848c/A of 5f2c97e00c770d329321cff2fbd6848c into e960ee71989d4b839dc250da7ef43fc9(size=12.4 K), total size for store is 24.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 02:55:50,186 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5f2c97e00c770d329321cff2fbd6848c: 2023-04-25 02:55:50,186 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c., storeName=5f2c97e00c770d329321cff2fbd6848c/A, priority=13, startTime=1682391349738; duration=0sec 2023-04-25 02:55:50,186 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 02:55:50,202 DEBUG [Thread-2900] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:55:50,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:55:50,205 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:55:50,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:55:50,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:55:50,206 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:55:50,206 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:50,206 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:55:50,206 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:55:50,207 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:50,207 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:50,209 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:50,209 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:50,209 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:50,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:50,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:50,209 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:50,210 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:55:50,210 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:50,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:55:50,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:50,210 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:55:50,210 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:55:50,211 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:55:50,211 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:55:50,211 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:50,211 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:55:50,211 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:55:50,211 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:55:50,211 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. 2023-04-25 02:55:50,211 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. started... 2023-04-25 02:55:50,211 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. as already flushing 2023-04-25 02:55:50,211 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. 2023-04-25 02:55:50,211 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:55:50,211 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:55:50,211 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:50,211 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:55:50,211 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:55:50,214 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:50,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:50,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:50,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:50,214 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:50,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:50,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:50,215 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:50,215 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:55:50,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:50,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:50,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:50,216 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:55:50,216 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@784ba4ec[Count = 0] remaining members to acquire global barrier 2023-04-25 02:55:50,216 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:55:50,216 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:50,217 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:50,218 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:55:50,218 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:50,218 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:50,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:50,218 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:55:50,218 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:55:50,218 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:55:50,220 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:55:50,220 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:50,220 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:55:50,221 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:50,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:50,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:50,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:50,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:50,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:50,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:50,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:50,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:50,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:50,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:50,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:55:50,224 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:55:50,224 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:55:50,224 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:55:50,224 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:55:50,224 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:55:50,224 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:55:50,226 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:50,227 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:50,227 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:50,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:50,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:50,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:50,227 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:50,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:50,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:50,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:50,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:50,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:50,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:50,229 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:50,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:50,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:50,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:50,230 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:50,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:50,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:50,231 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:50,233 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:50,233 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:50,233 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:50,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:50,234 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:50,234 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:50,234 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:50,234 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:50,234 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:50,234 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:50,234 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:50,234 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:50,234 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:55:50,234 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:55:50,234 DEBUG [Thread-2900] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:55:50,234 DEBUG [Thread-2900] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:55:50,234 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:50,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:50,334 DEBUG [Thread-2900] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:55:50,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:55:50,337 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:55:50,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:55:50,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:55:50,338 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:55:50,338 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:50,338 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:55:50,338 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:55:50,338 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:50,338 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:50,339 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:50,339 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:50,339 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:50,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:50,340 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:50,340 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:55:50,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:50,340 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:50,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:55:50,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:50,341 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:55:50,341 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:55:50,341 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:55:50,341 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:55:50,341 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:50,341 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:55:50,342 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:55:50,342 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:55:50,342 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. 2023-04-25 02:55:50,342 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. started... 2023-04-25 02:55:50,342 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. as already flushing 2023-04-25 02:55:50,342 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. 2023-04-25 02:55:50,342 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:55:50,342 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:55:50,342 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:50,342 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:55:50,342 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:55:50,342 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:50,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59162 deadline: 1682391410340, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:50,342 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:50,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59116 deadline: 1682391410341, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:50,345 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:50,345 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:50,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:50,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:50,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:50,345 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:50,345 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:55:50,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:50,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:50,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:50,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:50,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:50,347 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:55:50,347 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@56242d5c[Count = 0] remaining members to acquire global barrier 2023-04-25 02:55:50,347 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:55:50,347 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:50,348 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:50,348 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:50,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:50,348 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:55:50,348 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:55:50,348 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:55:50,348 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:50,348 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:55:50,349 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:50,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:50,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:50,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:50,349 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:55:50,349 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:50,350 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:55:50,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:50,351 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:50,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59136 deadline: 1682391410346, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:50,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:50,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:50,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:50,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:50,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:50,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:50,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:55:50,354 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:55:50,354 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:55:50,354 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:55:50,354 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:55:50,354 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:55:50,354 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:55:50,358 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:50,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:50,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:50,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:50,358 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:50,358 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:50,358 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:50,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:50,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:50,359 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:50,359 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:50,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:50,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:50,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:50,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:50,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:50,362 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:50,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:50,362 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=300 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp/A/d233cc3284d74a9986a80262b272c485 2023-04-25 02:55:50,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:50,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:50,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:50,366 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:50,366 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:50,366 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:50,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:50,367 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:50,367 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:50,367 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:50,367 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:50,367 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:50,367 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:50,367 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:50,367 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:50,367 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:50,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:50,367 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:55:50,367 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:55:50,367 DEBUG [Thread-2900] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:55:50,368 DEBUG [Thread-2900] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:55:50,368 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:55:50,468 DEBUG [Thread-2900] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:55:50,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:55:50,470 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:55:50,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:55:50,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:55:50,471 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:55:50,471 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:50,471 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:55:50,471 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:55:50,472 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:50,472 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:50,473 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:50,473 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:50,473 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:50,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:50,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:50,474 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:50,474 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:55:50,474 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:50,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:55:50,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:50,474 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:55:50,474 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:55:50,474 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:55:50,474 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:55:50,475 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:50,475 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:55:50,475 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:55:50,475 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:55:50,475 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. 2023-04-25 02:55:50,475 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. started... 2023-04-25 02:55:50,475 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. as already flushing 2023-04-25 02:55:50,475 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. 2023-04-25 02:55:50,475 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:55:50,475 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:55:50,475 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:50,475 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:55:50,475 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:55:50,477 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:50,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:50,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:50,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:50,477 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:50,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:50,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:50,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:50,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:50,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:50,478 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:50,479 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:55:50,479 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:55:50,479 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@22223b5a[Count = 0] remaining members to acquire global barrier 2023-04-25 02:55:50,479 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:55:50,479 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:50,481 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:50,481 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:50,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:50,481 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:55:50,481 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:55:50,481 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:55:50,482 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:50,482 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:55:50,483 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:55:50,483 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:50,483 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:55:50,483 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:50,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:50,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:50,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:50,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:50,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:50,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:50,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:50,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:50,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:50,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:50,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:55:50,486 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:55:50,487 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:55:50,487 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:55:50,487 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:55:50,487 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:55:50,487 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:55:50,489 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:50,489 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:50,489 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:50,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:50,489 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:50,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:50,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:50,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:50,489 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:50,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:50,490 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:50,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:50,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:50,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:50,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:50,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:50,491 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:50,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:50,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:50,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:50,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:50,495 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:50,495 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:50,495 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:50,495 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:50,495 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:50,495 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:50,495 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:50,495 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:55:50,495 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:55:50,495 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:50,495 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:55:50,495 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:50,495 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:50,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:50,495 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:50,495 DEBUG [Thread-2900] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:55:50,496 DEBUG [Thread-2900] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:55:50,497 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:50,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:50,596 DEBUG [Thread-2900] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:55:50,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:55:50,598 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:55:50,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:55:50,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:55:50,604 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:55:50,604 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:50,604 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:55:50,604 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:55:50,604 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:50,604 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:50,612 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:50,612 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:50,612 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:50,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:50,612 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:50,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:50,612 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:55:50,612 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:50,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:55:50,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:50,613 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:55:50,613 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:55:50,613 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:55:50,613 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:55:50,613 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:50,613 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:55:50,613 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:55:50,613 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:55:50,613 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. 2023-04-25 02:55:50,613 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. started... 2023-04-25 02:55:50,614 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. as already flushing 2023-04-25 02:55:50,614 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. 2023-04-25 02:55:50,614 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:55:50,614 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:55:50,614 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:50,614 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:55:50,614 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:55:50,616 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:50,616 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:50,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:50,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:50,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:50,616 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:50,616 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:55:50,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:50,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:50,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:50,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:50,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:50,617 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:55:50,617 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@412d1ca8[Count = 0] remaining members to acquire global barrier 2023-04-25 02:55:50,617 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:55:50,617 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:50,619 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:50,619 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:50,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:50,619 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:55:50,619 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:55:50,619 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:55:50,619 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:50,619 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:55:50,620 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:55:50,620 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:50,620 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:50,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:50,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:50,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:50,621 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:55:50,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:50,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:50,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:50,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:50,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:50,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:50,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:50,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:55:50,623 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:55:50,623 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:55:50,623 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:55:50,623 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:55:50,623 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:55:50,623 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:55:50,625 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:50,625 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:50,625 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:50,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:50,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:50,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:50,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:50,625 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:50,625 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:50,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:50,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:50,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:50,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:50,626 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:50,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:50,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:50,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:50,627 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:50,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:50,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:50,630 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:50,630 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:50,630 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:50,630 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:50,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:50,630 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:50,630 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:50,630 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:50,630 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:50,630 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:50,630 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:50,630 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:50,630 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:55:50,630 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:55:50,630 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:55:50,630 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:50,631 DEBUG [Thread-2900] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:55:50,631 DEBUG [Thread-2900] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:55:50,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:50,650 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:50,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59116 deadline: 1682391410645, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:50,651 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:50,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59162 deadline: 1682391410645, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:50,660 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:50,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59136 deadline: 1682391410653, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:50,731 DEBUG [Thread-2900] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:55:50,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:55:50,733 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:55:50,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:55:50,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:55:50,735 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:55:50,735 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:50,735 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:55:50,735 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:55:50,735 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:50,735 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:50,736 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:50,736 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:50,736 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:50,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:50,737 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:50,737 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:55:50,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:50,737 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:50,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:55:50,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:50,737 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:55:50,738 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:55:50,738 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:55:50,738 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:55:50,738 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:50,738 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:55:50,738 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:55:50,738 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:55:50,738 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. 2023-04-25 02:55:50,738 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. started... 2023-04-25 02:55:50,738 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. as already flushing 2023-04-25 02:55:50,738 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. 2023-04-25 02:55:50,738 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:55:50,738 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:55:50,739 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:50,739 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:55:50,739 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:55:50,740 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:50,740 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:50,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:50,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:50,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:50,741 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:50,741 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:55:50,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:50,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:50,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:50,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:50,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:50,746 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:55:50,746 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6464aca5[Count = 0] remaining members to acquire global barrier 2023-04-25 02:55:50,746 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:55:50,746 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:50,749 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:50,749 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:50,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:50,749 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:55:50,749 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:55:50,749 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:55:50,749 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:50,749 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:55:50,752 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:55:50,752 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:50,752 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:50,752 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:55:50,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:50,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:50,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:50,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:50,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:50,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:50,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:50,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:50,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:50,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:50,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:55:50,755 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:55:50,755 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:55:50,755 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:55:50,755 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:55:50,755 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:55:50,755 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:55:50,758 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:50,758 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:50,758 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:50,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:50,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:50,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:50,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:50,758 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:50,759 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:50,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:50,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:50,759 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:50,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:50,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:50,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:50,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:50,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:50,761 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:50,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:50,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:50,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:50,764 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:50,764 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:50,764 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:50,764 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:50,764 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:55:50,764 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:55:50,764 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:55:50,764 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:50,764 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:50,764 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:50,764 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:50,764 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:50,765 DEBUG [Thread-2900] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:55:50,765 DEBUG [Thread-2900] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:55:50,765 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:50,765 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:50,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:50,765 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:50,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:50,782 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=300 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp/B/5f55beaa53c541359b99496ee8f45d01 2023-04-25 02:55:50,810 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=300 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp/C/8b4d55c9537740638b13d13c4b4f4505 2023-04-25 02:55:50,815 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp/A/d233cc3284d74a9986a80262b272c485 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/d233cc3284d74a9986a80262b272c485 2023-04-25 02:55:50,818 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/d233cc3284d74a9986a80262b272c485, entries=200, sequenceid=300, filesize=14.3 K 2023-04-25 02:55:50,819 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp/B/5f55beaa53c541359b99496ee8f45d01 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/5f55beaa53c541359b99496ee8f45d01 2023-04-25 02:55:50,825 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/5f55beaa53c541359b99496ee8f45d01, entries=150, sequenceid=300, filesize=11.9 K 2023-04-25 02:55:50,826 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp/C/8b4d55c9537740638b13d13c4b4f4505 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/8b4d55c9537740638b13d13c4b4f4505 2023-04-25 02:55:50,829 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/8b4d55c9537740638b13d13c4b4f4505, entries=150, sequenceid=300, filesize=11.9 K 2023-04-25 02:55:50,829 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 5f2c97e00c770d329321cff2fbd6848c in 896ms, sequenceid=300, compaction requested=true 2023-04-25 02:55:50,830 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 5f2c97e00c770d329321cff2fbd6848c: 2023-04-25 02:55:50,830 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 02:55:50,830 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 02:55:50,830 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-25 02:55:50,830 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-25 02:55:50,830 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 02:55:50,831 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37054 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-25 02:55:50,831 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HStore(1912): 5f2c97e00c770d329321cff2fbd6848c/B is initiating minor compaction (all files) 2023-04-25 02:55:50,831 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39494 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-25 02:55:50,831 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 5f2c97e00c770d329321cff2fbd6848c/B in TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. 2023-04-25 02:55:50,831 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HStore(1912): 5f2c97e00c770d329321cff2fbd6848c/A is initiating minor compaction (all files) 2023-04-25 02:55:50,831 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/56d61f70a2d4416ab32bef05bd696a21, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/9a6aa4970f514d45a22dbd79b8707a39, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/5f55beaa53c541359b99496ee8f45d01] into tmpdir=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp, totalSize=36.2 K 2023-04-25 02:55:50,831 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 5f2c97e00c770d329321cff2fbd6848c/A in TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. 2023-04-25 02:55:50,831 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/e960ee71989d4b839dc250da7ef43fc9, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/29dc053dd8cf4baca8fb1bcc48973535, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/d233cc3284d74a9986a80262b272c485] into tmpdir=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp, totalSize=38.6 K 2023-04-25 02:55:50,832 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting 56d61f70a2d4416ab32bef05bd696a21, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=270, earliestPutTs=1682391347670 2023-04-25 02:55:50,832 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting e960ee71989d4b839dc250da7ef43fc9, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=270, earliestPutTs=1682391347670 2023-04-25 02:55:50,832 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting 9a6aa4970f514d45a22dbd79b8707a39, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=287, earliestPutTs=1682391349269 2023-04-25 02:55:50,832 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting 29dc053dd8cf4baca8fb1bcc48973535, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=287, earliestPutTs=1682391349269 2023-04-25 02:55:50,832 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting 5f55beaa53c541359b99496ee8f45d01, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=300, earliestPutTs=1682391349917 2023-04-25 02:55:50,833 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting d233cc3284d74a9986a80262b272c485, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=300, earliestPutTs=1682391349917 2023-04-25 02:55:50,838 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] throttle.PressureAwareThroughputController(145): 5f2c97e00c770d329321cff2fbd6848c#B#compaction#432 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-25 02:55:50,841 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] throttle.PressureAwareThroughputController(145): 5f2c97e00c770d329321cff2fbd6848c#A#compaction#433 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-25 02:55:50,865 DEBUG [Thread-2900] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:55:50,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:55:50,866 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp/A/3ea22b53818540b59d1577f0406582ac as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/3ea22b53818540b59d1577f0406582ac 2023-04-25 02:55:50,867 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:55:50,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:55:50,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:55:50,868 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:55:50,868 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:50,868 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:55:50,869 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:55:50,871 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:50,872 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:50,873 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 5f2c97e00c770d329321cff2fbd6848c/A of 5f2c97e00c770d329321cff2fbd6848c into 3ea22b53818540b59d1577f0406582ac(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 02:55:50,873 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5f2c97e00c770d329321cff2fbd6848c: 2023-04-25 02:55:50,873 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c., storeName=5f2c97e00c770d329321cff2fbd6848c/A, priority=13, startTime=1682391350830; duration=0sec 2023-04-25 02:55:50,873 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 02:55:50,874 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-25 02:55:50,874 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37054 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-25 02:55:50,874 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HStore(1912): 5f2c97e00c770d329321cff2fbd6848c/C is initiating minor compaction (all files) 2023-04-25 02:55:50,874 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 5f2c97e00c770d329321cff2fbd6848c/C in TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. 2023-04-25 02:55:50,875 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:50,875 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:50,875 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:50,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:50,875 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/7f506986abab4b5089b3c8f0ddcc4400, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/c27b7be3b60c421ca54880277c0f73ce, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/8b4d55c9537740638b13d13c4b4f4505] into tmpdir=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp, totalSize=36.2 K 2023-04-25 02:55:50,876 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting 7f506986abab4b5089b3c8f0ddcc4400, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=270, earliestPutTs=1682391347670 2023-04-25 02:55:50,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:50,877 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:50,877 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:55:50,878 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting c27b7be3b60c421ca54880277c0f73ce, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=287, earliestPutTs=1682391349269 2023-04-25 02:55:50,878 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:50,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:55:50,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:50,880 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting 8b4d55c9537740638b13d13c4b4f4505, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=300, earliestPutTs=1682391349917 2023-04-25 02:55:50,881 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:55:50,881 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:55:50,881 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:55:50,881 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:55:50,881 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:50,881 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:55:50,884 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:55:50,884 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:55:50,885 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. 2023-04-25 02:55:50,885 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. started... 2023-04-25 02:55:50,886 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5f2c97e00c770d329321cff2fbd6848c 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-25 02:55:50,886 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5f2c97e00c770d329321cff2fbd6848c, store=A 2023-04-25 02:55:50,886 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:55:50,886 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5f2c97e00c770d329321cff2fbd6848c, store=B 2023-04-25 02:55:50,886 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:55:50,886 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5f2c97e00c770d329321cff2fbd6848c, store=C 2023-04-25 02:55:50,886 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:55:50,897 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] throttle.PressureAwareThroughputController(145): 5f2c97e00c770d329321cff2fbd6848c#C#compaction#434 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-25 02:55:50,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(9158): Flush requested on 5f2c97e00c770d329321cff2fbd6848c 2023-04-25 02:55:50,944 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. as already flushing 2023-04-25 02:55:50,972 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:50,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59114 deadline: 1682391410967, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:50,982 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:50,983 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59156 deadline: 1682391410974, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:51,079 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:51,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59114 deadline: 1682391411074, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:51,089 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:51,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59156 deadline: 1682391411085, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:51,157 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:51,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59116 deadline: 1682391411153, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:51,162 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:51,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59162 deadline: 1682391411156, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:51,164 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:51,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59136 deadline: 1682391411163, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:51,258 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp/B/0a4a8c56432f458a8eb20f3d505e7eec as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/0a4a8c56432f458a8eb20f3d505e7eec 2023-04-25 02:55:51,263 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 5f2c97e00c770d329321cff2fbd6848c/B of 5f2c97e00c770d329321cff2fbd6848c into 0a4a8c56432f458a8eb20f3d505e7eec(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 02:55:51,263 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5f2c97e00c770d329321cff2fbd6848c: 2023-04-25 02:55:51,263 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c., storeName=5f2c97e00c770d329321cff2fbd6848c/B, priority=13, startTime=1682391350830; duration=0sec 2023-04-25 02:55:51,263 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 02:55:51,285 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:51,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59114 deadline: 1682391411282, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:51,298 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:51,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59156 deadline: 1682391411291, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:51,322 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=322 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp/A/02a2d8e212ba4bac80d6c320de09f911 2023-04-25 02:55:51,334 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp/C/a925bc96190f4e5dbddaf31b5c850ca0 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/a925bc96190f4e5dbddaf31b5c850ca0 2023-04-25 02:55:51,336 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=322 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp/B/41af84bba07341a1856955d04d58cf9b 2023-04-25 02:55:51,339 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 5f2c97e00c770d329321cff2fbd6848c/C of 5f2c97e00c770d329321cff2fbd6848c into a925bc96190f4e5dbddaf31b5c850ca0(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 02:55:51,339 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5f2c97e00c770d329321cff2fbd6848c: 2023-04-25 02:55:51,339 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c., storeName=5f2c97e00c770d329321cff2fbd6848c/C, priority=13, startTime=1682391350830; duration=0sec 2023-04-25 02:55:51,340 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 02:55:51,368 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=322 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp/C/c0eb8b5035e94230963c2369fe6d38e9 2023-04-25 02:55:51,372 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp/A/02a2d8e212ba4bac80d6c320de09f911 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/02a2d8e212ba4bac80d6c320de09f911 2023-04-25 02:55:51,377 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/02a2d8e212ba4bac80d6c320de09f911, entries=150, sequenceid=322, filesize=11.9 K 2023-04-25 02:55:51,378 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp/B/41af84bba07341a1856955d04d58cf9b as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/41af84bba07341a1856955d04d58cf9b 2023-04-25 02:55:51,384 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/41af84bba07341a1856955d04d58cf9b, entries=150, sequenceid=322, filesize=11.9 K 2023-04-25 02:55:51,385 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp/C/c0eb8b5035e94230963c2369fe6d38e9 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/c0eb8b5035e94230963c2369fe6d38e9 2023-04-25 02:55:51,388 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/c0eb8b5035e94230963c2369fe6d38e9, entries=150, sequenceid=322, filesize=11.9 K 2023-04-25 02:55:51,389 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 5f2c97e00c770d329321cff2fbd6848c in 502ms, sequenceid=322, compaction requested=false 2023-04-25 02:55:51,389 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5f2c97e00c770d329321cff2fbd6848c: 2023-04-25 02:55:51,389 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. 2023-04-25 02:55:51,389 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:55:51,389 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:55:51,389 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:51,389 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:55:51,389 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:55:51,390 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:51,390 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:51,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:51,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:51,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:51,390 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:51,390 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:55:51,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:51,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:51,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:51,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:51,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:51,392 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:55:51,392 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@135d33b8[Count = 0] remaining members to acquire global barrier 2023-04-25 02:55:51,392 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:55:51,392 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:51,396 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:51,396 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:51,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:51,396 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:55:51,396 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:55:51,396 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:55:51,396 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:51,396 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:55:51,398 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:55:51,398 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:51,398 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:51,398 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:55:51,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:51,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:51,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:51,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:51,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:51,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:51,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:51,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:51,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:51,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:51,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:55:51,401 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:55:51,401 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:55:51,401 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:55:51,401 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:55:51,401 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:55:51,401 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:55:51,408 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:51,408 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:51,408 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:51,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:51,408 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:51,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:51,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:51,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:51,408 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:51,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:51,409 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:51,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:51,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:51,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:51,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:51,411 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:51,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:51,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:51,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:51,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:51,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:51,414 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:51,414 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:51,414 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:51,414 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:51,414 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:51,414 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:55:51,414 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:55:51,414 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:51,414 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:51,415 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:51,415 DEBUG [Thread-2900] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:55:51,415 DEBUG [Thread-2900] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:55:51,415 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:55:51,416 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:51,416 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:51,416 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:51,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:51,416 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:51,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:51,515 DEBUG [Thread-2900] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:55:51,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:55:51,517 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:55:51,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:55:51,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:55:51,518 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:55:51,518 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:51,518 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:55:51,518 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:55:51,519 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:51,519 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:51,520 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:51,520 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:51,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:51,520 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:51,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:51,521 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:51,521 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:55:51,521 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:51,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:55:51,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:51,521 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:55:51,521 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:55:51,521 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:55:51,521 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:55:51,522 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:51,522 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:55:51,522 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:55:51,522 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:55:51,522 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. 2023-04-25 02:55:51,522 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. started... 2023-04-25 02:55:51,522 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5f2c97e00c770d329321cff2fbd6848c 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-25 02:55:51,522 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5f2c97e00c770d329321cff2fbd6848c, store=A 2023-04-25 02:55:51,523 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:55:51,523 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5f2c97e00c770d329321cff2fbd6848c, store=B 2023-04-25 02:55:51,523 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:55:51,523 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5f2c97e00c770d329321cff2fbd6848c, store=C 2023-04-25 02:55:51,523 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:55:51,546 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=336 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp/A/11d6d204318f4c4c87a006b0f44a5a17 2023-04-25 02:55:51,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(9158): Flush requested on 5f2c97e00c770d329321cff2fbd6848c 2023-04-25 02:55:51,592 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. as already flushing 2023-04-25 02:55:51,601 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=336 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp/B/f681a8228f19424f8e43a7548ab53972 2023-04-25 02:55:51,635 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=336 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp/C/80a67d4136f04b3a9560ad8185535640 2023-04-25 02:55:51,639 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp/A/11d6d204318f4c4c87a006b0f44a5a17 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/11d6d204318f4c4c87a006b0f44a5a17 2023-04-25 02:55:51,644 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/11d6d204318f4c4c87a006b0f44a5a17, entries=150, sequenceid=336, filesize=11.9 K 2023-04-25 02:55:51,644 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp/B/f681a8228f19424f8e43a7548ab53972 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/f681a8228f19424f8e43a7548ab53972 2023-04-25 02:55:51,648 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/f681a8228f19424f8e43a7548ab53972, entries=150, sequenceid=336, filesize=11.9 K 2023-04-25 02:55:51,649 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp/C/80a67d4136f04b3a9560ad8185535640 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/80a67d4136f04b3a9560ad8185535640 2023-04-25 02:55:51,658 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/80a67d4136f04b3a9560ad8185535640, entries=150, sequenceid=336, filesize=11.9 K 2023-04-25 02:55:51,658 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=100.63 KB/103050 for 5f2c97e00c770d329321cff2fbd6848c in 136ms, sequenceid=336, compaction requested=true 2023-04-25 02:55:51,659 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5f2c97e00c770d329321cff2fbd6848c: 2023-04-25 02:55:51,659 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. 2023-04-25 02:55:51,659 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:55:51,659 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:55:51,659 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:51,660 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:55:51,660 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:55:51,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(9158): Flush requested on 5f2c97e00c770d329321cff2fbd6848c 2023-04-25 02:55:51,665 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 5f2c97e00c770d329321cff2fbd6848c 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-25 02:55:51,665 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5f2c97e00c770d329321cff2fbd6848c, store=A 2023-04-25 02:55:51,665 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:55:51,665 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5f2c97e00c770d329321cff2fbd6848c, store=B 2023-04-25 02:55:51,665 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:55:51,665 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5f2c97e00c770d329321cff2fbd6848c, store=C 2023-04-25 02:55:51,665 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:55:51,666 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:51,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:51,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:51,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:51,666 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:51,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:51,667 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:51,667 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:55:51,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:51,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:51,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:51,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:51,673 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:55:51,673 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@20163262[Count = 0] remaining members to acquire global barrier 2023-04-25 02:55:51,673 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:55:51,673 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:51,674 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:51,674 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:51,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:51,674 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:55:51,674 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:55:51,674 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:55:51,675 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:51,675 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:55:51,676 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:51,676 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:55:51,676 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:51,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:51,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:51,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:51,676 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:55:51,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:51,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:51,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:51,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:51,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:51,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:51,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:51,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:55:51,679 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:55:51,679 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:55:51,679 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:55:51,679 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:55:51,679 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:55:51,679 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:55:51,682 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:51,682 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:51,682 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:51,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:51,682 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:51,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:51,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:51,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:51,684 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:51,684 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:51,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:51,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:51,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:51,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:51,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:51,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:51,685 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:51,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:51,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:51,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:51,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:51,692 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:51,692 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:51,693 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:51,693 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:51,693 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:51,693 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:51,693 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:55:51,693 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:55:51,693 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:51,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:51,693 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=355 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp/A/c99f0d64b1574017ae53b1efa60850da 2023-04-25 02:55:51,693 DEBUG [Thread-2900] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:55:51,693 DEBUG [Thread-2900] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:55:51,693 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:51,693 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:55:51,693 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:51,694 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:51,694 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:51,694 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:51,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:51,726 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=355 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp/B/6216c66b71c74e74ac217622b6c8b5eb 2023-04-25 02:55:51,731 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:51,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59114 deadline: 1682391411720, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:51,732 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:51,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59156 deadline: 1682391411721, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:51,745 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=355 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp/C/b3275707397d42c1aed222e5ec36ff2c 2023-04-25 02:55:51,751 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp/A/c99f0d64b1574017ae53b1efa60850da as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/c99f0d64b1574017ae53b1efa60850da 2023-04-25 02:55:51,754 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/c99f0d64b1574017ae53b1efa60850da, entries=200, sequenceid=355, filesize=14.3 K 2023-04-25 02:55:51,759 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp/B/6216c66b71c74e74ac217622b6c8b5eb as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/6216c66b71c74e74ac217622b6c8b5eb 2023-04-25 02:55:51,763 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/6216c66b71c74e74ac217622b6c8b5eb, entries=150, sequenceid=355, filesize=11.9 K 2023-04-25 02:55:51,763 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp/C/b3275707397d42c1aed222e5ec36ff2c as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/b3275707397d42c1aed222e5ec36ff2c 2023-04-25 02:55:51,770 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/b3275707397d42c1aed222e5ec36ff2c, entries=150, sequenceid=355, filesize=11.9 K 2023-04-25 02:55:51,771 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 5f2c97e00c770d329321cff2fbd6848c in 106ms, sequenceid=355, compaction requested=true 2023-04-25 02:55:51,771 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 5f2c97e00c770d329321cff2fbd6848c: 2023-04-25 02:55:51,771 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 02:55:51,771 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-25 02:55:51,771 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-25 02:55:51,771 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 02:55:51,771 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 02:55:51,774 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49392 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-25 02:55:51,774 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HStore(1912): 5f2c97e00c770d329321cff2fbd6848c/B is initiating minor compaction (all files) 2023-04-25 02:55:51,774 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 51832 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-25 02:55:51,774 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 5f2c97e00c770d329321cff2fbd6848c/B in TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. 2023-04-25 02:55:51,774 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HStore(1912): 5f2c97e00c770d329321cff2fbd6848c/A is initiating minor compaction (all files) 2023-04-25 02:55:51,774 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/0a4a8c56432f458a8eb20f3d505e7eec, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/41af84bba07341a1856955d04d58cf9b, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/f681a8228f19424f8e43a7548ab53972, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/6216c66b71c74e74ac217622b6c8b5eb] into tmpdir=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp, totalSize=48.2 K 2023-04-25 02:55:51,774 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 5f2c97e00c770d329321cff2fbd6848c/A in TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. 2023-04-25 02:55:51,774 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/3ea22b53818540b59d1577f0406582ac, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/02a2d8e212ba4bac80d6c320de09f911, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/11d6d204318f4c4c87a006b0f44a5a17, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/c99f0d64b1574017ae53b1efa60850da] into tmpdir=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp, totalSize=50.6 K 2023-04-25 02:55:51,775 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting 0a4a8c56432f458a8eb20f3d505e7eec, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=300, earliestPutTs=1682391349917 2023-04-25 02:55:51,776 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting 3ea22b53818540b59d1577f0406582ac, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=300, earliestPutTs=1682391349917 2023-04-25 02:55:51,776 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting 02a2d8e212ba4bac80d6c320de09f911, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=322, earliestPutTs=1682391350010 2023-04-25 02:55:51,776 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting 41af84bba07341a1856955d04d58cf9b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=322, earliestPutTs=1682391350010 2023-04-25 02:55:51,777 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting f681a8228f19424f8e43a7548ab53972, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=336, earliestPutTs=1682391350956 2023-04-25 02:55:51,777 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting 11d6d204318f4c4c87a006b0f44a5a17, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=336, earliestPutTs=1682391350956 2023-04-25 02:55:51,777 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting 6216c66b71c74e74ac217622b6c8b5eb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=355, earliestPutTs=1682391351625 2023-04-25 02:55:51,777 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting c99f0d64b1574017ae53b1efa60850da, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=355, earliestPutTs=1682391351625 2023-04-25 02:55:51,793 DEBUG [Thread-2900] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:55:51,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:55:51,796 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:55:51,798 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] throttle.PressureAwareThroughputController(145): 5f2c97e00c770d329321cff2fbd6848c#B#compaction#444 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-25 02:55:51,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:55:51,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:55:51,808 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:55:51,808 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:51,808 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:55:51,808 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:55:51,809 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:51,809 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:51,810 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:51,810 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:51,810 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:51,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:51,810 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:51,810 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:55:51,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:51,811 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:51,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:55:51,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:51,811 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:55:51,822 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:55:51,822 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:55:51,822 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:55:51,822 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:51,822 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:55:51,822 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:55:51,822 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:55:51,824 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] throttle.PressureAwareThroughputController(145): 5f2c97e00c770d329321cff2fbd6848c#A#compaction#445 average throughput is 1.64 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-25 02:55:51,824 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. 2023-04-25 02:55:51,824 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. started... 2023-04-25 02:55:51,825 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5f2c97e00c770d329321cff2fbd6848c 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-25 02:55:51,825 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5f2c97e00c770d329321cff2fbd6848c, store=A 2023-04-25 02:55:51,825 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:55:51,825 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5f2c97e00c770d329321cff2fbd6848c, store=B 2023-04-25 02:55:51,825 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:55:51,825 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5f2c97e00c770d329321cff2fbd6848c, store=C 2023-04-25 02:55:51,825 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:55:51,842 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp/B/5ecbe96417ed42cf98ef2e791b9f9e80 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/5ecbe96417ed42cf98ef2e791b9f9e80 2023-04-25 02:55:51,845 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(9158): Flush requested on 5f2c97e00c770d329321cff2fbd6848c 2023-04-25 02:55:51,845 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. as already flushing 2023-04-25 02:55:51,848 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 5f2c97e00c770d329321cff2fbd6848c/B of 5f2c97e00c770d329321cff2fbd6848c into 5ecbe96417ed42cf98ef2e791b9f9e80(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 02:55:51,848 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5f2c97e00c770d329321cff2fbd6848c: 2023-04-25 02:55:51,848 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c., storeName=5f2c97e00c770d329321cff2fbd6848c/B, priority=12, startTime=1682391351771; duration=0sec 2023-04-25 02:55:51,848 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 02:55:51,848 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-25 02:55:51,850 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49392 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-25 02:55:51,850 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HStore(1912): 5f2c97e00c770d329321cff2fbd6848c/C is initiating minor compaction (all files) 2023-04-25 02:55:51,850 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 5f2c97e00c770d329321cff2fbd6848c/C in TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. 2023-04-25 02:55:51,850 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/a925bc96190f4e5dbddaf31b5c850ca0, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/c0eb8b5035e94230963c2369fe6d38e9, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/80a67d4136f04b3a9560ad8185535640, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/b3275707397d42c1aed222e5ec36ff2c] into tmpdir=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp, totalSize=48.2 K 2023-04-25 02:55:51,851 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting a925bc96190f4e5dbddaf31b5c850ca0, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=300, earliestPutTs=1682391349917 2023-04-25 02:55:51,853 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting c0eb8b5035e94230963c2369fe6d38e9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=322, earliestPutTs=1682391350010 2023-04-25 02:55:51,854 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting 80a67d4136f04b3a9560ad8185535640, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=336, earliestPutTs=1682391350956 2023-04-25 02:55:51,854 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting b3275707397d42c1aed222e5ec36ff2c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=355, earliestPutTs=1682391351625 2023-04-25 02:55:51,870 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=368 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp/A/bb2b08510e834f1abdcf36eb3b31e8ee 2023-04-25 02:55:51,873 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp/A/ef11df27c39f451e8247b0342327f248 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/ef11df27c39f451e8247b0342327f248 2023-04-25 02:55:51,879 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 5f2c97e00c770d329321cff2fbd6848c/A of 5f2c97e00c770d329321cff2fbd6848c into ef11df27c39f451e8247b0342327f248(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 02:55:51,879 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5f2c97e00c770d329321cff2fbd6848c: 2023-04-25 02:55:51,879 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c., storeName=5f2c97e00c770d329321cff2fbd6848c/A, priority=12, startTime=1682391351771; duration=0sec 2023-04-25 02:55:51,879 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 02:55:51,881 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] throttle.PressureAwareThroughputController(145): 5f2c97e00c770d329321cff2fbd6848c#C#compaction#447 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-25 02:55:51,900 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp/C/89d7b0aeae6448e7a70466fca51e6eb3 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/89d7b0aeae6448e7a70466fca51e6eb3 2023-04-25 02:55:51,904 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 5f2c97e00c770d329321cff2fbd6848c/C of 5f2c97e00c770d329321cff2fbd6848c into 89d7b0aeae6448e7a70466fca51e6eb3(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 02:55:51,904 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5f2c97e00c770d329321cff2fbd6848c: 2023-04-25 02:55:51,904 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c., storeName=5f2c97e00c770d329321cff2fbd6848c/C, priority=12, startTime=1682391351771; duration=0sec 2023-04-25 02:55:51,904 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 02:55:51,920 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:51,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59156 deadline: 1682391411914, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:51,924 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:51,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59114 deadline: 1682391411920, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:52,025 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:52,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59156 deadline: 1682391412021, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:52,032 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:52,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59114 deadline: 1682391412026, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:52,163 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:52,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59116 deadline: 1682391412158, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:52,172 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:52,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59136 deadline: 1682391412167, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:52,172 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:52,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59162 deadline: 1682391412169, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:52,230 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:52,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59156 deadline: 1682391412227, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:52,238 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:52,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59114 deadline: 1682391412234, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:52,296 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=368 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp/B/51b07b94467b421bbeaeba2d38e2afe1 2023-04-25 02:55:52,310 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=368 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp/C/482ed7ef431846318b4100732febde8a 2023-04-25 02:55:52,316 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp/A/bb2b08510e834f1abdcf36eb3b31e8ee as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/bb2b08510e834f1abdcf36eb3b31e8ee 2023-04-25 02:55:52,320 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/bb2b08510e834f1abdcf36eb3b31e8ee, entries=150, sequenceid=368, filesize=11.9 K 2023-04-25 02:55:52,321 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp/B/51b07b94467b421bbeaeba2d38e2afe1 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/51b07b94467b421bbeaeba2d38e2afe1 2023-04-25 02:55:52,332 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/51b07b94467b421bbeaeba2d38e2afe1, entries=150, sequenceid=368, filesize=11.9 K 2023-04-25 02:55:52,333 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp/C/482ed7ef431846318b4100732febde8a as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/482ed7ef431846318b4100732febde8a 2023-04-25 02:55:52,336 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/482ed7ef431846318b4100732febde8a, entries=150, sequenceid=368, filesize=11.9 K 2023-04-25 02:55:52,337 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 5f2c97e00c770d329321cff2fbd6848c in 512ms, sequenceid=368, compaction requested=false 2023-04-25 02:55:52,337 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5f2c97e00c770d329321cff2fbd6848c: 2023-04-25 02:55:52,337 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. 2023-04-25 02:55:52,337 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:55:52,337 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:55:52,337 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:52,337 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:55:52,337 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:55:52,341 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:52,341 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:52,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:52,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:52,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:52,342 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:52,342 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:55:52,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:52,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:52,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:52,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:52,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:52,343 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:55:52,343 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@977a294[Count = 0] remaining members to acquire global barrier 2023-04-25 02:55:52,343 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:55:52,343 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:52,344 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:52,344 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:52,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:52,345 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:55:52,345 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:55:52,345 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:55:52,345 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:52,345 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:55:52,347 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:55:52,347 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:52,347 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:55:52,347 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:52,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:52,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:52,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:52,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:52,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:52,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:52,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:52,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:52,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:52,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:52,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:55:52,350 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:55:52,350 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:55:52,350 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:55:52,350 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:55:52,350 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:55:52,350 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:55:52,352 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:52,352 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:52,352 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:52,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:52,352 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:52,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:52,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:52,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:52,352 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:52,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:52,353 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:52,353 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:52,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:52,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:52,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:52,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:52,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:52,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:52,358 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:52,358 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:52,358 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:52,358 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:52,358 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:52,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:52,358 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:52,358 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:52,358 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:52,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:52,358 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:52,358 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:52,359 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:52,359 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:55:52,359 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:55:52,359 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:52,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:52,359 DEBUG [Thread-2900] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:55:52,359 DEBUG [Thread-2900] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:55:52,359 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:55:52,459 DEBUG [Thread-2900] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:55:52,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:55:52,461 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:55:52,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:55:52,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:55:52,463 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:55:52,463 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:52,463 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:55:52,463 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:55:52,464 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:52,464 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:52,465 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:52,465 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:52,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:52,465 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:52,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:52,465 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:52,465 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:55:52,466 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:52,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:55:52,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:52,466 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:55:52,466 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:55:52,466 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:55:52,467 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:55:52,467 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:52,467 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:55:52,467 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:55:52,467 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:55:52,467 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. 2023-04-25 02:55:52,467 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. started... 2023-04-25 02:55:52,467 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5f2c97e00c770d329321cff2fbd6848c 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-25 02:55:52,467 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5f2c97e00c770d329321cff2fbd6848c, store=A 2023-04-25 02:55:52,468 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:55:52,468 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5f2c97e00c770d329321cff2fbd6848c, store=B 2023-04-25 02:55:52,468 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:55:52,468 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5f2c97e00c770d329321cff2fbd6848c, store=C 2023-04-25 02:55:52,468 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:55:52,494 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=390 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp/A/05c380d9cee547bc85d140c88cb8ec3a 2023-04-25 02:55:52,535 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(9158): Flush requested on 5f2c97e00c770d329321cff2fbd6848c 2023-04-25 02:55:52,535 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. as already flushing 2023-04-25 02:55:52,571 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:52,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59114 deadline: 1682391412563, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:52,574 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:52,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59156 deadline: 1682391412564, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:52,677 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:52,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59114 deadline: 1682391412673, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:52,680 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:52,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59156 deadline: 1682391412675, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:52,884 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:52,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59114 deadline: 1682391412880, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:52,885 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:52,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59156 deadline: 1682391412881, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:52,905 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=390 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp/B/e5a8d5db6d424e00b954889567b1e122 2023-04-25 02:55:52,916 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=390 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp/C/361b6a9994fb47e8a3eca9a1973a0e6f 2023-04-25 02:55:52,922 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp/A/05c380d9cee547bc85d140c88cb8ec3a as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/05c380d9cee547bc85d140c88cb8ec3a 2023-04-25 02:55:52,926 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/05c380d9cee547bc85d140c88cb8ec3a, entries=150, sequenceid=390, filesize=11.9 K 2023-04-25 02:55:52,927 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp/B/e5a8d5db6d424e00b954889567b1e122 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/e5a8d5db6d424e00b954889567b1e122 2023-04-25 02:55:52,931 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/e5a8d5db6d424e00b954889567b1e122, entries=150, sequenceid=390, filesize=11.9 K 2023-04-25 02:55:52,932 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp/C/361b6a9994fb47e8a3eca9a1973a0e6f as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/361b6a9994fb47e8a3eca9a1973a0e6f 2023-04-25 02:55:52,935 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/361b6a9994fb47e8a3eca9a1973a0e6f, entries=150, sequenceid=390, filesize=11.9 K 2023-04-25 02:55:52,936 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 5f2c97e00c770d329321cff2fbd6848c in 469ms, sequenceid=390, compaction requested=true 2023-04-25 02:55:52,936 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5f2c97e00c770d329321cff2fbd6848c: 2023-04-25 02:55:52,936 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. 2023-04-25 02:55:52,936 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:55:52,936 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:55:52,936 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:52,936 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:55:52,936 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:55:52,939 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:52,939 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:52,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:52,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:52,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:52,939 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:52,939 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:55:52,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:52,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:52,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:52,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:52,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:52,941 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:55:52,941 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@48b7369b[Count = 0] remaining members to acquire global barrier 2023-04-25 02:55:52,941 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:55:52,941 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:52,942 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:52,942 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:52,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:52,942 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:55:52,942 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:55:52,942 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:55:52,942 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:52,942 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:55:52,945 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:55:52,945 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:52,945 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:55:52,945 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:52,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:52,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:52,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:52,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:52,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:52,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:52,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:52,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:52,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:52,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:52,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:55:52,947 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:55:52,947 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:55:52,947 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:55:52,947 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:55:52,947 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:55:52,947 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:55:52,948 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:52,949 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:52,949 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:52,949 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:52,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:52,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:52,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:52,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:52,949 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:52,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:52,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:52,949 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:52,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:52,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:52,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:52,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:52,952 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:52,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:52,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:52,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:52,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:52,964 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:52,964 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:52,964 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:52,964 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:52,964 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:52,964 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:52,964 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:52,964 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:52,964 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:55:52,964 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:55:52,965 DEBUG [Thread-2900] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:55:52,965 DEBUG [Thread-2900] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:55:52,965 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:52,965 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:52,965 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:52,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:52,966 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:52,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:53,065 DEBUG [Thread-2900] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:55:53,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:55:53,067 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:55:53,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:55:53,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:55:53,069 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:55:53,069 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:53,069 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:55:53,069 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:55:53,069 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:53,069 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:53,071 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:53,071 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:53,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:53,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:53,072 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:53,072 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:53,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:55:53,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:53,072 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:55:53,073 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:55:53,073 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:55:53,073 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:55:53,073 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:53,073 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:55:53,073 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:53,073 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:55:53,073 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:55:53,073 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:55:53,073 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. 2023-04-25 02:55:53,073 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. started... 2023-04-25 02:55:53,074 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5f2c97e00c770d329321cff2fbd6848c 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-25 02:55:53,074 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5f2c97e00c770d329321cff2fbd6848c, store=A 2023-04-25 02:55:53,074 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:55:53,074 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5f2c97e00c770d329321cff2fbd6848c, store=B 2023-04-25 02:55:53,074 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:55:53,074 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5f2c97e00c770d329321cff2fbd6848c, store=C 2023-04-25 02:55:53,074 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:55:53,091 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=403 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp/A/f4efb953219346e786755552924893ea 2023-04-25 02:55:53,130 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=403 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp/B/3c58268d9c214b2eb17a686707f58b4b 2023-04-25 02:55:53,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(9158): Flush requested on 5f2c97e00c770d329321cff2fbd6848c 2023-04-25 02:55:53,192 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. as already flushing 2023-04-25 02:55:53,253 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:53,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59114 deadline: 1682391413247, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:53,254 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:53,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59156 deadline: 1682391413247, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:53,358 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:53,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59114 deadline: 1682391413354, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:53,359 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:53,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59156 deadline: 1682391413356, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:53,543 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=403 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp/C/99ff5b002c1e42429cab7e3a978368ba 2023-04-25 02:55:53,546 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp/A/f4efb953219346e786755552924893ea as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/f4efb953219346e786755552924893ea 2023-04-25 02:55:53,549 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/f4efb953219346e786755552924893ea, entries=150, sequenceid=403, filesize=11.9 K 2023-04-25 02:55:53,550 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp/B/3c58268d9c214b2eb17a686707f58b4b as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/3c58268d9c214b2eb17a686707f58b4b 2023-04-25 02:55:53,553 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/3c58268d9c214b2eb17a686707f58b4b, entries=150, sequenceid=403, filesize=11.9 K 2023-04-25 02:55:53,554 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp/C/99ff5b002c1e42429cab7e3a978368ba as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/99ff5b002c1e42429cab7e3a978368ba 2023-04-25 02:55:53,557 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/99ff5b002c1e42429cab7e3a978368ba, entries=150, sequenceid=403, filesize=11.9 K 2023-04-25 02:55:53,558 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 5f2c97e00c770d329321cff2fbd6848c in 485ms, sequenceid=403, compaction requested=true 2023-04-25 02:55:53,558 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5f2c97e00c770d329321cff2fbd6848c: 2023-04-25 02:55:53,558 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. 2023-04-25 02:55:53,558 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:55:53,558 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:55:53,558 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:53,558 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:55:53,558 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:55:53,560 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:53,560 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:53,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:53,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:53,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:53,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:53,560 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:53,560 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:55:53,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:53,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:53,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:53,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:53,561 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:55:53,561 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@320d0cc6[Count = 0] remaining members to acquire global barrier 2023-04-25 02:55:53,561 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:55:53,561 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:53,564 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:53,564 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:53,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:53,564 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:55:53,564 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:55:53,564 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:55:53,564 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:53,564 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:55:53,565 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:53,565 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:55:53,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:53,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:53,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:53,565 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:53,565 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:55:53,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:53,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:53,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:53,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:53,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:53,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:53,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:53,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:55:53,569 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:55:53,569 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:55:53,569 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:55:53,569 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:55:53,569 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682391353236, max timestamp=1682391353567Num uniques 859; 2023-04-25 02:55:53,569 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:55:53,569 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-25 02:55:53,569 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:55:53,569 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682391353236, max timestamp=1682391353567Num uniques 859; 2023-04-25 02:55:53,569 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-25 02:55:53,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(9158): Flush requested on 5f2c97e00c770d329321cff2fbd6848c 2023-04-25 02:55:53,570 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 5f2c97e00c770d329321cff2fbd6848c 3/3 column families, dataSize=62.62 KB heapSize=174.33 KB 2023-04-25 02:55:53,570 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5f2c97e00c770d329321cff2fbd6848c, store=A 2023-04-25 02:55:53,570 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 02:55:53,570 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5f2c97e00c770d329321cff2fbd6848c, store=B 2023-04-25 02:55:53,570 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 02:55:53,570 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5f2c97e00c770d329321cff2fbd6848c, store=C 2023-04-25 02:55:53,570 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 02:55:53,571 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:53,572 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:53,572 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:53,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:53,571 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:53,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:53,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:53,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:53,572 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:53,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:53,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:53,573 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:53,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:53,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:53,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:53,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:53,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:53,574 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:53,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:53,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:53,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:53,576 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:53,576 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:53,576 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:53,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:53,576 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:53,576 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:55:53,576 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:53,576 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:55:53,576 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:55:53,576 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:53,576 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:53,577 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:53,577 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:53,577 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:53,577 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:53,577 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:53,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:53,577 DEBUG [Thread-2900] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:55:53,577 DEBUG [Thread-2900] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:55:53,578 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=424 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp/A/0fad8006f82a41f8864f9dd51fe3abe0 2023-04-25 02:55:53,594 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=424 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp/B/8343c71db4ae4de0967cd6a6977c3604 2023-04-25 02:55:53,612 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=424 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp/C/4e6b3a559fc949b3bbee45b745e6fa56 2023-04-25 02:55:53,616 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp/A/0fad8006f82a41f8864f9dd51fe3abe0 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/0fad8006f82a41f8864f9dd51fe3abe0 2023-04-25 02:55:53,620 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/0fad8006f82a41f8864f9dd51fe3abe0, entries=250, sequenceid=424, filesize=16.7 K 2023-04-25 02:55:53,620 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp/B/8343c71db4ae4de0967cd6a6977c3604 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/8343c71db4ae4de0967cd6a6977c3604 2023-04-25 02:55:53,624 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/8343c71db4ae4de0967cd6a6977c3604, entries=150, sequenceid=424, filesize=11.9 K 2023-04-25 02:55:53,625 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp/C/4e6b3a559fc949b3bbee45b745e6fa56 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/4e6b3a559fc949b3bbee45b745e6fa56 2023-04-25 02:55:53,629 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/4e6b3a559fc949b3bbee45b745e6fa56, entries=150, sequenceid=424, filesize=11.9 K 2023-04-25 02:55:53,629 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~62.62 KB/64120, heapSize ~174.28 KB/178464, currentSize=80.51 KB/82440 for 5f2c97e00c770d329321cff2fbd6848c in 59ms, sequenceid=424, compaction requested=true 2023-04-25 02:55:53,629 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 5f2c97e00c770d329321cff2fbd6848c: 2023-04-25 02:55:53,630 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 02:55:53,630 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 02:55:53,630 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-25 02:55:53,630 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-25 02:55:53,630 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 02:55:53,631 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 66594 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-25 02:55:53,631 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61714 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-25 02:55:53,631 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HStore(1912): 5f2c97e00c770d329321cff2fbd6848c/A is initiating minor compaction (all files) 2023-04-25 02:55:53,631 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HStore(1912): 5f2c97e00c770d329321cff2fbd6848c/B is initiating minor compaction (all files) 2023-04-25 02:55:53,631 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 5f2c97e00c770d329321cff2fbd6848c/A in TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. 2023-04-25 02:55:53,631 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 5f2c97e00c770d329321cff2fbd6848c/B in TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. 2023-04-25 02:55:53,631 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/ef11df27c39f451e8247b0342327f248, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/bb2b08510e834f1abdcf36eb3b31e8ee, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/05c380d9cee547bc85d140c88cb8ec3a, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/f4efb953219346e786755552924893ea, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/0fad8006f82a41f8864f9dd51fe3abe0] into tmpdir=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp, totalSize=65.0 K 2023-04-25 02:55:53,631 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/5ecbe96417ed42cf98ef2e791b9f9e80, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/51b07b94467b421bbeaeba2d38e2afe1, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/e5a8d5db6d424e00b954889567b1e122, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/3c58268d9c214b2eb17a686707f58b4b, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/8343c71db4ae4de0967cd6a6977c3604] into tmpdir=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp, totalSize=60.3 K 2023-04-25 02:55:53,632 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting 5ecbe96417ed42cf98ef2e791b9f9e80, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=355, earliestPutTs=1682391351625 2023-04-25 02:55:53,632 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting ef11df27c39f451e8247b0342327f248, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=355, earliestPutTs=1682391351625 2023-04-25 02:55:53,633 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting 51b07b94467b421bbeaeba2d38e2afe1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=368, earliestPutTs=1682391351696 2023-04-25 02:55:53,633 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting bb2b08510e834f1abdcf36eb3b31e8ee, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=368, earliestPutTs=1682391351696 2023-04-25 02:55:53,633 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting e5a8d5db6d424e00b954889567b1e122, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=390, earliestPutTs=1682391351892 2023-04-25 02:55:53,633 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting 05c380d9cee547bc85d140c88cb8ec3a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=390, earliestPutTs=1682391351892 2023-04-25 02:55:53,633 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting 3c58268d9c214b2eb17a686707f58b4b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=403, earliestPutTs=1682391352549 2023-04-25 02:55:53,633 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting f4efb953219346e786755552924893ea, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=403, earliestPutTs=1682391352549 2023-04-25 02:55:53,634 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting 8343c71db4ae4de0967cd6a6977c3604, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=424, earliestPutTs=1682391353236 2023-04-25 02:55:53,634 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting 0fad8006f82a41f8864f9dd51fe3abe0, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=424, earliestPutTs=1682391353236 2023-04-25 02:55:53,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(9158): Flush requested on 5f2c97e00c770d329321cff2fbd6848c 2023-04-25 02:55:53,637 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 5f2c97e00c770d329321cff2fbd6848c 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-25 02:55:53,637 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5f2c97e00c770d329321cff2fbd6848c, store=A 2023-04-25 02:55:53,637 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:55:53,637 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5f2c97e00c770d329321cff2fbd6848c, store=B 2023-04-25 02:55:53,637 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:55:53,637 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5f2c97e00c770d329321cff2fbd6848c, store=C 2023-04-25 02:55:53,637 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:55:53,649 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] throttle.PressureAwareThroughputController(145): 5f2c97e00c770d329321cff2fbd6848c#B#compaction#460 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-25 02:55:53,653 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] throttle.PressureAwareThroughputController(145): 5f2c97e00c770d329321cff2fbd6848c#A#compaction#461 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-25 02:55:53,677 DEBUG [Thread-2900] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:55:53,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:55:53,679 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:55:53,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:55:53,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:55:53,683 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:55:53,684 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:53,684 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:55:53,684 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:55:53,687 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:53,688 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:53,690 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:53,690 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:53,690 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:53,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:53,691 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:53,691 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:55:53,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:53,691 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp/B/7ea43f735d6641d39ac3a4b33c99abf6 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/7ea43f735d6641d39ac3a4b33c99abf6 2023-04-25 02:55:53,692 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:53,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:55:53,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:53,693 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:55:53,694 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:55:53,694 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:55:53,694 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:55:53,694 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:53,694 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:55:53,694 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:55:53,694 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:55:53,694 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. 2023-04-25 02:55:53,694 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. started... 2023-04-25 02:55:53,695 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. as already flushing 2023-04-25 02:55:53,695 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. 2023-04-25 02:55:53,695 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:55:53,695 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:55:53,695 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:53,695 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:55:53,695 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:55:53,695 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp/A/66eef2882bba40208d1637b2b78094c7 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/66eef2882bba40208d1637b2b78094c7 2023-04-25 02:55:53,697 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:53,697 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:53,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:53,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:53,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:53,698 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 5f2c97e00c770d329321cff2fbd6848c/B of 5f2c97e00c770d329321cff2fbd6848c into 7ea43f735d6641d39ac3a4b33c99abf6(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 02:55:53,698 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5f2c97e00c770d329321cff2fbd6848c: 2023-04-25 02:55:53,698 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c., storeName=5f2c97e00c770d329321cff2fbd6848c/B, priority=11, startTime=1682391353630; duration=0sec 2023-04-25 02:55:53,698 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 02:55:53,698 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-25 02:55:53,700 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61714 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-25 02:55:53,700 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HStore(1912): 5f2c97e00c770d329321cff2fbd6848c/C is initiating minor compaction (all files) 2023-04-25 02:55:53,700 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 5f2c97e00c770d329321cff2fbd6848c/C in TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. 2023-04-25 02:55:53,700 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/89d7b0aeae6448e7a70466fca51e6eb3, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/482ed7ef431846318b4100732febde8a, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/361b6a9994fb47e8a3eca9a1973a0e6f, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/99ff5b002c1e42429cab7e3a978368ba, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/4e6b3a559fc949b3bbee45b745e6fa56] into tmpdir=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp, totalSize=60.3 K 2023-04-25 02:55:53,700 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 5f2c97e00c770d329321cff2fbd6848c/A of 5f2c97e00c770d329321cff2fbd6848c into 66eef2882bba40208d1637b2b78094c7(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 02:55:53,700 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:53,701 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:55:53,701 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting 89d7b0aeae6448e7a70466fca51e6eb3, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=355, earliestPutTs=1682391351625 2023-04-25 02:55:53,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:53,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:53,700 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5f2c97e00c770d329321cff2fbd6848c: 2023-04-25 02:55:53,701 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c., storeName=5f2c97e00c770d329321cff2fbd6848c/A, priority=11, startTime=1682391353630; duration=0sec 2023-04-25 02:55:53,701 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting 482ed7ef431846318b4100732febde8a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=368, earliestPutTs=1682391351696 2023-04-25 02:55:53,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:53,701 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 02:55:53,702 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting 361b6a9994fb47e8a3eca9a1973a0e6f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=390, earliestPutTs=1682391351892 2023-04-25 02:55:53,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:53,702 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting 99ff5b002c1e42429cab7e3a978368ba, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=403, earliestPutTs=1682391352549 2023-04-25 02:55:53,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:53,702 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting 4e6b3a559fc949b3bbee45b745e6fa56, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=424, earliestPutTs=1682391353236 2023-04-25 02:55:53,702 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:55:53,702 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2319adc7[Count = 0] remaining members to acquire global barrier 2023-04-25 02:55:53,702 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:55:53,702 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:53,704 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:53,704 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:53,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:53,704 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:55:53,704 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:55:53,704 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:55:53,704 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:53,704 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:55:53,705 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:53,705 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:55:53,705 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:53,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:53,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:53,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:53,706 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:55:53,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:53,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:53,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:53,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:53,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:53,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:53,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:53,708 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:53,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59156 deadline: 1682391413700, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:53,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:55:53,708 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:55:53,708 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:55:53,708 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:55:53,708 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:55:53,708 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:55:53,708 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:55:53,713 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:53,713 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:53,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:53,713 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:53,713 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:53,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:53,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:53,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:53,713 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:53,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:53,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:53,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:53,714 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:53,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:53,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:53,714 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] throttle.PressureAwareThroughputController(145): 5f2c97e00c770d329321cff2fbd6848c#C#compaction#462 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-25 02:55:53,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:53,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:53,715 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:53,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:53,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:53,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:53,731 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:53,731 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:53,731 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:53,731 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:53,732 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:55:53,732 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:55:53,732 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:53,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:53,732 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:53,732 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:55:53,731 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:53,731 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:53,732 DEBUG [Thread-2900] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:55:53,732 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:53,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:53,732 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:53,732 DEBUG [Thread-2900] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:55:53,732 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:53,732 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:53,734 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:53,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59114 deadline: 1682391413705, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:53,817 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:53,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59156 deadline: 1682391413809, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:53,832 DEBUG [Thread-2900] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:55:53,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:55:53,834 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:55:53,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:55:53,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:55:53,836 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:55:53,836 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:53,836 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:55:53,836 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:55:53,836 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:53,836 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:53,837 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:53,837 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:53,838 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:53,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:53,838 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:53,838 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:55:53,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:53,838 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:53,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:55:53,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:53,838 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:55:53,839 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:55:53,839 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:55:53,839 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:55:53,839 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:53,839 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:55:53,839 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:55:53,839 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:55:53,839 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. 2023-04-25 02:55:53,839 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. started... 2023-04-25 02:55:53,839 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. as already flushing 2023-04-25 02:55:53,839 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. 2023-04-25 02:55:53,839 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:55:53,839 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:55:53,840 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:53,840 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:55:53,840 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:55:53,842 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:53,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59114 deadline: 1682391413836, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:53,842 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:53,842 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:53,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:53,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:53,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:53,842 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:53,842 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:55:53,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:53,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:53,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:53,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:53,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:53,844 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:55:53,844 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@174ed832[Count = 0] remaining members to acquire global barrier 2023-04-25 02:55:53,844 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:55:53,844 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:53,847 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:53,847 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:53,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:53,847 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:55:53,847 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:55:53,847 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:55:53,847 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:53,847 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:55:53,849 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:55:53,849 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:53,849 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:55:53,850 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:53,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:53,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:53,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:53,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:53,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:53,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:53,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:53,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:53,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:53,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:53,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:55:53,853 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:55:53,853 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:55:53,853 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:55:53,853 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:55:53,853 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:55:53,853 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:55:53,854 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:53,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:53,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:53,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:53,854 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:53,855 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:53,855 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:53,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:53,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:53,855 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:53,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:53,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:53,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:53,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:53,857 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:53,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:53,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:53,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:53,864 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:55:53,865 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:53,865 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:53,865 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:53,865 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:53,865 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:53,865 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:53,865 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:53,865 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:53,865 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:53,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:53,865 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:53,865 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:55:53,865 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:55:53,865 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:53,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:53,866 DEBUG [Thread-2900] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:55:53,866 DEBUG [Thread-2900] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:55:53,966 DEBUG [Thread-2900] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:55:53,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:55:53,968 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:55:53,970 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:55:53,970 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:55:53,970 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:55:53,970 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:53,970 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:55:53,970 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:55:53,971 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:53,971 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:53,972 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:53,972 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:53,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:53,972 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:53,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:53,973 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:53,973 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:55:53,973 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:53,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:55:53,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:53,974 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:55:53,974 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:55:53,974 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:55:53,974 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:55:53,974 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:53,974 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:55:53,979 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:55:53,979 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:55:53,979 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. 2023-04-25 02:55:53,979 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. started... 2023-04-25 02:55:53,979 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. as already flushing 2023-04-25 02:55:53,979 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. 2023-04-25 02:55:53,979 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:55:53,979 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:55:53,979 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:53,979 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:55:53,979 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:55:53,981 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:53,981 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:53,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:53,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:53,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:53,981 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:53,981 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:55:53,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:53,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:53,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:53,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:53,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:53,984 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:55:53,984 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@66bc27ae[Count = 0] remaining members to acquire global barrier 2023-04-25 02:55:53,984 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:55:53,984 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:53,986 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:53,986 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:53,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:53,987 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:55:53,987 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:55:53,987 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:55:53,987 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:53,987 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:55:53,988 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:55:53,988 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:53,988 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:53,988 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:55:53,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:53,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:53,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:53,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:53,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:53,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:53,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:53,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:53,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:53,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:53,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:55:53,991 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:55:53,991 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:55:53,991 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:55:53,991 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:55:53,991 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:55:53,991 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:55:53,994 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:53,994 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:53,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:53,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:53,994 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:53,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:53,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:53,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:53,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:53,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:53,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:53,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:53,995 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:53,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:53,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:53,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:53,996 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:53,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:53,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:53,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:53,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:53,999 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:53,999 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:53,999 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:53,999 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:53,999 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:53,999 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:53,999 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:53,999 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:53,999 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:53,999 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:54,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:54,000 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:54,000 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:55:54,000 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:55:54,000 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:54,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:54,000 DEBUG [Thread-2900] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:55:54,000 DEBUG [Thread-2900] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:55:54,000 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:55:54,022 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:54,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59156 deadline: 1682391414018, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:54,047 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:54,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59114 deadline: 1682391414043, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:54,078 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=440 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp/A/9e2fa0e41ba64e1782832107abb9d0ac 2023-04-25 02:55:54,091 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=440 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp/B/a8603cbb821b4b3ba43793987f38a2fc 2023-04-25 02:55:54,100 DEBUG [Thread-2900] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:55:54,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:55:54,102 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:55:54,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:55:54,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:55:54,106 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:55:54,106 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:54,106 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:55:54,106 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:55:54,106 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:54,106 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:54,109 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:54,109 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:54,109 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:54,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:54,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:54,110 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:54,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:55:54,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:54,110 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:55:54,110 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:54,110 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:55:54,116 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:55:54,116 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:55:54,116 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:55:54,116 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:54,116 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:55:54,121 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:55:54,121 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:55:54,121 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. 2023-04-25 02:55:54,121 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. started... 2023-04-25 02:55:54,121 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. as already flushing 2023-04-25 02:55:54,121 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. 2023-04-25 02:55:54,121 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:55:54,121 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:55:54,121 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:54,121 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:55:54,121 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:55:54,123 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=440 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp/C/8d281d48a1a241a290c8e810179fd82c 2023-04-25 02:55:54,124 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:54,124 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:54,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:54,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:54,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:54,124 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:54,124 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:55:54,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:54,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:54,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:54,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:54,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:54,126 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:55:54,126 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3c8d7c83[Count = 0] remaining members to acquire global barrier 2023-04-25 02:55:54,126 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:55:54,126 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:54,127 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:54,127 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:54,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:54,128 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:55:54,128 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:55:54,128 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:55:54,128 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp/A/9e2fa0e41ba64e1782832107abb9d0ac as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/9e2fa0e41ba64e1782832107abb9d0ac 2023-04-25 02:55:54,128 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:54,128 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:55:54,129 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:54,129 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:55:54,129 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:54,129 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:55:54,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:54,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:54,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:54,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:54,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:54,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:54,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:54,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:54,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:54,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:54,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:55:54,132 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:55:54,132 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:55:54,132 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:55:54,132 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:55:54,132 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:55:54,132 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:55:54,134 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/9e2fa0e41ba64e1782832107abb9d0ac, entries=200, sequenceid=440, filesize=14.3 K 2023-04-25 02:55:54,135 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp/B/a8603cbb821b4b3ba43793987f38a2fc as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/a8603cbb821b4b3ba43793987f38a2fc 2023-04-25 02:55:54,135 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:54,135 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:54,135 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:54,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:54,136 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:54,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:54,136 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:54,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:54,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:54,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:54,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:54,137 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:54,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:54,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:54,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:54,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:54,138 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:54,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:54,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:54,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:54,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:54,141 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/a8603cbb821b4b3ba43793987f38a2fc, entries=150, sequenceid=440, filesize=11.9 K 2023-04-25 02:55:54,141 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:54,141 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:54,141 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:54,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:54,142 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:54,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:54,142 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:54,142 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:54,142 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:54,142 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:54,142 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:54,142 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:54,142 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:55:54,142 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:55:54,142 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:54,142 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:54,142 DEBUG [Thread-2900] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:55:54,143 DEBUG [Thread-2900] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:55:54,143 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp/C/8d281d48a1a241a290c8e810179fd82c as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/8d281d48a1a241a290c8e810179fd82c 2023-04-25 02:55:54,143 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:55:54,146 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp/C/1b75821bb43b4cf78fb3f5fcf43a3a7b as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/1b75821bb43b4cf78fb3f5fcf43a3a7b 2023-04-25 02:55:54,148 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/8d281d48a1a241a290c8e810179fd82c, entries=150, sequenceid=440, filesize=11.9 K 2023-04-25 02:55:54,149 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for 5f2c97e00c770d329321cff2fbd6848c in 512ms, sequenceid=440, compaction requested=false 2023-04-25 02:55:54,149 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 5f2c97e00c770d329321cff2fbd6848c: 2023-04-25 02:55:54,152 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 5f2c97e00c770d329321cff2fbd6848c/C of 5f2c97e00c770d329321cff2fbd6848c into 1b75821bb43b4cf78fb3f5fcf43a3a7b(size=12.8 K), total size for store is 24.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 02:55:54,152 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5f2c97e00c770d329321cff2fbd6848c: 2023-04-25 02:55:54,152 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c., storeName=5f2c97e00c770d329321cff2fbd6848c/C, priority=11, startTime=1682391353630; duration=0sec 2023-04-25 02:55:54,152 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 02:55:54,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(9158): Flush requested on 5f2c97e00c770d329321cff2fbd6848c 2023-04-25 02:55:54,168 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 5f2c97e00c770d329321cff2fbd6848c 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-25 02:55:54,168 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5f2c97e00c770d329321cff2fbd6848c, store=A 2023-04-25 02:55:54,168 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:55:54,168 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5f2c97e00c770d329321cff2fbd6848c, store=B 2023-04-25 02:55:54,168 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:55:54,168 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5f2c97e00c770d329321cff2fbd6848c, store=C 2023-04-25 02:55:54,168 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:55:54,195 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=460 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp/A/f7bc9d8bdc0b4454950c0f53439b3276 2023-04-25 02:55:54,218 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:54,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59116 deadline: 1682391414210, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:54,226 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:54,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59136 deadline: 1682391414217, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:54,227 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:54,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59162 deadline: 1682391414218, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:54,243 DEBUG [Thread-2900] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:55:54,243 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:55:54,245 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:55:54,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:55:54,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:55:54,246 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:55:54,246 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:54,246 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:55:54,246 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:55:54,246 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:54,246 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:54,248 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:54,248 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:54,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:54,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:54,249 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:54,251 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:54,251 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:54,251 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:55:54,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:55:54,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:54,252 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:55:54,252 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:55:54,252 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:55:54,252 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:55:54,252 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:54,252 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:55:54,256 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:55:54,256 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:55:54,256 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. 2023-04-25 02:55:54,256 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. started... 2023-04-25 02:55:54,257 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. as already flushing 2023-04-25 02:55:54,257 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. 2023-04-25 02:55:54,257 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:55:54,257 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:55:54,257 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:54,257 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:55:54,257 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:55:54,258 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:54,258 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:54,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:54,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:54,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:54,259 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:54,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:54,259 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:55:54,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:54,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:54,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:54,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:54,260 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:55:54,260 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@52949940[Count = 0] remaining members to acquire global barrier 2023-04-25 02:55:54,260 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:55:54,260 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:54,263 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:54,263 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:54,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:54,263 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:55:54,263 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:55:54,263 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:55:54,263 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:54,263 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:55:54,264 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:54,264 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:55:54,264 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:54,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:54,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:54,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:54,265 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:55:54,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:54,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:54,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:54,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:54,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:54,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:54,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:54,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:55:54,269 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:55:54,269 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:55:54,269 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:55:54,269 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:55:54,269 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:55:54,269 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:55:54,270 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:54,270 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:54,270 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:54,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:54,270 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:54,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:54,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:54,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:54,270 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:54,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:54,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:54,271 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:54,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:54,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:54,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:54,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:54,272 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:54,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:54,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:54,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:54,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:54,274 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:54,274 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:54,274 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:54,274 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:55:54,274 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:55:54,274 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:54,274 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:54,274 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:54,274 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:54,275 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:54,275 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:54,275 DEBUG [Thread-2900] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:55:54,275 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:54,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:54,275 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:54,275 DEBUG [Thread-2900] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:55:54,275 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:54,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:54,324 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:54,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59116 deadline: 1682391414319, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:54,333 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:54,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59156 deadline: 1682391414325, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:54,334 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:54,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59136 deadline: 1682391414327, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:54,334 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:54,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59162 deadline: 1682391414328, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:54,355 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:54,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59114 deadline: 1682391414349, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:54,375 DEBUG [Thread-2900] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:55:54,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:55:54,377 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:55:54,378 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:55:54,378 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:55:54,378 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:55:54,378 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:54,378 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:55:54,378 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:55:54,379 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:54,379 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:54,381 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:54,381 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:54,381 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:54,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:54,382 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:54,382 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:55:54,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:54,382 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:54,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:55:54,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:54,383 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:55:54,383 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:55:54,384 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:55:54,384 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:55:54,384 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:54,384 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:55:54,385 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:55:54,385 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:55:54,385 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. 2023-04-25 02:55:54,385 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. started... 2023-04-25 02:55:54,385 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. as already flushing 2023-04-25 02:55:54,385 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. 2023-04-25 02:55:54,385 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:55:54,385 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:55:54,385 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:54,385 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:55:54,385 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:55:54,389 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:54,389 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:54,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:54,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:54,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:54,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:54,389 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:54,389 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:55:54,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:54,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:54,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:54,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:54,392 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:55:54,392 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@33fa5468[Count = 0] remaining members to acquire global barrier 2023-04-25 02:55:54,392 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:55:54,392 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:54,393 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:54,393 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:54,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:54,394 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:55:54,394 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:55:54,394 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:55:54,394 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:54,394 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:55:54,395 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:55:54,395 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:54,395 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:55:54,395 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:54,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:54,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:54,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:54,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:54,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:54,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:54,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:54,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:54,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:54,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:54,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:55:54,399 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:55:54,399 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:55:54,399 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:55:54,399 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:55:54,399 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:55:54,399 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:55:54,407 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:54,407 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:54,407 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:54,407 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:54,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:54,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:54,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:54,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:54,407 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:54,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:54,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:54,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:54,408 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:54,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:54,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:54,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:54,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:54,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:54,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:54,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:54,412 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:54,414 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:54,414 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:54,414 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:54,414 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:54,414 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:54,414 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:55:54,414 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:55:54,414 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:54,414 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:54,414 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:54,414 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:54,414 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:54,414 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:54,414 DEBUG [Thread-2900] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:55:54,414 DEBUG [Thread-2900] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:55:54,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:54,415 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:54,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:54,515 DEBUG [Thread-2900] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:55:54,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:55:54,517 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:55:54,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:55:54,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:55:54,518 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:55:54,518 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:54,518 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:55:54,518 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:55:54,519 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:54,519 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:54,520 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:54,520 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:54,520 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:54,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:54,520 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:54,520 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:55:54,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:54,521 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:54,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:55:54,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:54,521 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:55:54,522 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:55:54,522 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:55:54,522 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:55:54,522 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:54,522 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:55:54,522 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:55:54,522 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:55:54,522 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. 2023-04-25 02:55:54,522 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. started... 2023-04-25 02:55:54,522 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. as already flushing 2023-04-25 02:55:54,522 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. 2023-04-25 02:55:54,522 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:55:54,522 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:55:54,522 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:54,522 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:55:54,522 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:55:54,524 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:54,524 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:54,524 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:55:54,525 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:54,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:54,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:54,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:54,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:54,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:54,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:54,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:54,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:54,527 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:55:54,527 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@34fc94ee[Count = 0] remaining members to acquire global barrier 2023-04-25 02:55:54,527 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:55:54,527 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:54,529 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:54,529 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:54,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59116 deadline: 1682391414525, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:54,529 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:54,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:54,529 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:55:54,529 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:55:54,529 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:55:54,530 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:55:54,531 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:54,531 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:55:54,531 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:54,531 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:55:54,531 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:55:54,531 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:55:54,531 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:55:54,531 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:55:54,531 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:55:54,531 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:55:54,532 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:54,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:54,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:54,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:54,533 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:54,533 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:54,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:54,533 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:54,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:54,535 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:54,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:54,535 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:54,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:54,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:54,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:54,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:54,537 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:54,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:54,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:54,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:54,539 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:54,539 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:54,539 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:54,539 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:55:54,539 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:55:54,539 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:54,539 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:54,539 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:54,539 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:54,539 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:54,542 DEBUG [Thread-2900] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:55:54,542 DEBUG [Thread-2900] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:55:54,542 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:54,542 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:54,542 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:54,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:54,542 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:54,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:54,543 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:54,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59136 deadline: 1682391414536, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:54,543 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:54,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59162 deadline: 1682391414536, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:54,545 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:55:54,616 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=460 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp/B/662e556e867b4b4f92e9573ee399ab90 2023-04-25 02:55:54,642 DEBUG [Thread-2900] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:55:54,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:55:54,644 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:55:54,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:55:54,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:55:54,645 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:55:54,645 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:54,645 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:55:54,645 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:55:54,645 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:54,645 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:54,647 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:54,647 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:54,647 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:54,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:54,648 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:54,648 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:55:54,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:54,649 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:54,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:55:54,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:54,650 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:55:54,650 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:55:54,650 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:55:54,650 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:55:54,650 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:54,650 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:55:54,651 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:55:54,651 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:55:54,651 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. 2023-04-25 02:55:54,651 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. started... 2023-04-25 02:55:54,651 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. as already flushing 2023-04-25 02:55:54,651 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. 2023-04-25 02:55:54,651 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:55:54,651 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:55:54,651 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:54,651 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:55:54,651 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:55:54,653 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:54,653 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:54,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:54,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:54,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:54,653 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:54,653 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:55:54,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:54,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:54,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:54,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:54,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:54,654 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:55:54,654 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@63c29f96[Count = 0] remaining members to acquire global barrier 2023-04-25 02:55:54,654 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:55:54,654 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:54,656 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:54,656 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:54,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:54,656 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:55:54,656 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:55:54,656 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:54,656 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:55:54,656 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:55:54,657 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:54,657 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:55:54,657 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:54,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:54,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:54,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:54,657 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:55:54,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:54,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:54,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:54,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:54,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:54,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:54,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:54,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:55:54,659 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:55:54,659 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:55:54,660 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:55:54,660 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:55:54,660 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:55:54,660 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:55:54,662 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:54,663 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:54,663 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:54,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:54,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:54,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:54,663 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:54,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:54,663 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:54,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:54,663 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:54,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:54,663 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:54,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:54,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:54,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:54,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:54,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:54,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:54,666 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:54,666 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:54,666 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:54,666 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:54,666 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:54,666 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:54,666 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:54,666 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:54,666 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:54,666 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:55:54,666 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:55:54,666 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:54,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:54,666 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:55:54,666 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:54,666 DEBUG [Thread-2900] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:55:54,666 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:54,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:54,666 DEBUG [Thread-2900] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:55:54,767 DEBUG [Thread-2900] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:55:54,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:55:54,769 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:55:54,770 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:55:54,770 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:55:54,770 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:55:54,770 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:54,771 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:55:54,771 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:55:54,771 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:54,771 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:54,773 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:54,773 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:54,773 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:54,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:54,773 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:54,773 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:55:54,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:54,774 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:54,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:55:54,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:54,774 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:55:54,774 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:55:54,774 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:55:54,774 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:55:54,774 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:54,774 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:55:54,775 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:55:54,775 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:55:54,775 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. 2023-04-25 02:55:54,775 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. started... 2023-04-25 02:55:54,775 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. as already flushing 2023-04-25 02:55:54,775 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. 2023-04-25 02:55:54,775 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:55:54,775 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:55:54,775 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:54,775 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:55:54,775 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:55:54,777 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:54,777 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:54,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:54,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:54,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:54,777 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:54,777 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:55:54,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:54,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:54,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:54,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:54,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:54,778 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:55:54,779 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@156a5437[Count = 0] remaining members to acquire global barrier 2023-04-25 02:55:54,779 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:55:54,779 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:54,783 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:54,783 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:54,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:54,784 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:55:54,784 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:55:54,784 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:55:54,784 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:54,784 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:55:54,786 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:55:54,786 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:54,786 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:54,786 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:55:54,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:54,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:54,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:54,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:54,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:54,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:54,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:54,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:54,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:54,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:54,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:55:54,789 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:55:54,789 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:55:54,789 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:55:54,789 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:55:54,789 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:55:54,789 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:55:54,792 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:54,792 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:54,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:54,793 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:54,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:54,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:54,793 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:54,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:54,793 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:54,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:54,793 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:54,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:54,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:54,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:54,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:54,795 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:54,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:54,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:54,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:54,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:54,799 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:54,799 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:54,799 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:54,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:54,799 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:54,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:54,800 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:54,800 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:54,800 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:54,800 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:54,800 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:54,800 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:54,800 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:54,800 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:54,800 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:55:54,800 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:55:54,800 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:55:54,800 DEBUG [Thread-2900] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:55:54,800 DEBUG [Thread-2900] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:55:54,834 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:54,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59116 deadline: 1682391414831, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:54,839 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:54,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 201 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59156 deadline: 1682391414834, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:54,851 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:54,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59136 deadline: 1682391414844, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:54,851 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:54,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59162 deadline: 1682391414845, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:54,863 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:54,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59114 deadline: 1682391414859, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:54,900 DEBUG [Thread-2900] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:55:54,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:55:54,903 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:55:54,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:55:54,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:55:54,904 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:55:54,904 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:54,904 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:55:54,904 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:55:54,905 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:54,905 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:54,906 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:54,906 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:54,906 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:54,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:54,906 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:54,906 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:55:54,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:54,907 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:54,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:55:54,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:54,907 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:55:54,907 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:55:54,907 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:55:54,907 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:55:54,907 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:54,908 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:55:54,908 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:55:54,908 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:55:54,908 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. 2023-04-25 02:55:54,908 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. started... 2023-04-25 02:55:54,908 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. as already flushing 2023-04-25 02:55:54,908 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. 2023-04-25 02:55:54,908 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:55:54,908 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:55:54,908 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:54,908 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:55:54,908 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:55:54,910 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:54,910 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:54,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:54,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:54,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:54,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:54,910 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:54,910 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:55:54,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:54,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:54,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:54,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:54,912 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:55:54,912 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@74520be6[Count = 0] remaining members to acquire global barrier 2023-04-25 02:55:54,912 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:55:54,912 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:54,913 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:54,913 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:54,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:54,913 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:55:54,914 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:55:54,914 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:55:54,914 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:54,914 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:55:54,915 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:55:54,916 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:54,916 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:55:54,916 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:54,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:54,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:54,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:54,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:54,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:54,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:54,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:54,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:54,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:54,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:54,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:55:54,920 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:55:54,920 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:55:54,920 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:55:54,920 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:55:54,920 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:55:54,920 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:55:54,922 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:54,922 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:54,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:54,922 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:54,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:54,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:54,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:54,922 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:54,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:54,922 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:54,923 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:54,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:54,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:54,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:54,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:54,924 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:54,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:54,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:54,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:54,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:54,926 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:54,926 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:54,926 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:54,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:54,926 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:54,926 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:54,926 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:54,926 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:54,926 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:54,926 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:54,926 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:54,926 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:54,926 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:55:54,926 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:55:54,926 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:55:54,927 DEBUG [Thread-2900] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:55:54,927 DEBUG [Thread-2900] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:55:54,927 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:54,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:55,027 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=460 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp/C/9e7e07dea075439aabcb5d0ca4df3923 2023-04-25 02:55:55,027 DEBUG [Thread-2900] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:55:55,028 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:55:55,029 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:55:55,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:55:55,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:55:55,031 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:55:55,031 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:55,031 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:55:55,031 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:55:55,031 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp/A/f7bc9d8bdc0b4454950c0f53439b3276 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/f7bc9d8bdc0b4454950c0f53439b3276 2023-04-25 02:55:55,031 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:55,031 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:55,033 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:55,033 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:55,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:55,033 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:55,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:55,033 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:55,033 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:55:55,033 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:55,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:55:55,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:55,034 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:55:55,034 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:55:55,034 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:55:55,034 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:55:55,034 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:55,034 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:55:55,036 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:55:55,036 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:55:55,036 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. 2023-04-25 02:55:55,036 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. started... 2023-04-25 02:55:55,036 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. as already flushing 2023-04-25 02:55:55,036 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. 2023-04-25 02:55:55,036 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/f7bc9d8bdc0b4454950c0f53439b3276, entries=200, sequenceid=460, filesize=14.3 K 2023-04-25 02:55:55,036 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:55:55,036 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:55:55,036 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:55,036 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:55:55,036 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:55:55,037 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp/B/662e556e867b4b4f92e9573ee399ab90 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/662e556e867b4b4f92e9573ee399ab90 2023-04-25 02:55:55,039 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:55,039 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:55,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:55,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:55,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:55,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:55,039 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:55,039 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:55:55,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:55,040 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/662e556e867b4b4f92e9573ee399ab90, entries=150, sequenceid=460, filesize=11.9 K 2023-04-25 02:55:55,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:55,042 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp/C/9e7e07dea075439aabcb5d0ca4df3923 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/9e7e07dea075439aabcb5d0ca4df3923 2023-04-25 02:55:55,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:55,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:55,044 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:55:55,044 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@24d9aa66[Count = 0] remaining members to acquire global barrier 2023-04-25 02:55:55,044 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:55:55,044 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:55,045 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:55,045 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:55,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:55,045 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:55:55,045 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:55:55,045 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:55:55,045 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:55,045 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:55:55,046 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/9e7e07dea075439aabcb5d0ca4df3923, entries=150, sequenceid=460, filesize=11.9 K 2023-04-25 02:55:55,046 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for 5f2c97e00c770d329321cff2fbd6848c in 878ms, sequenceid=460, compaction requested=true 2023-04-25 02:55:55,046 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:55,047 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 5f2c97e00c770d329321cff2fbd6848c: 2023-04-25 02:55:55,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:55,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:55,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:55,047 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 02:55:55,047 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-25 02:55:55,047 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-25 02:55:55,047 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:55:55,047 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:55,047 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 02:55:55,047 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:55:55,047 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 02:55:55,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:55,048 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 42392 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-25 02:55:55,048 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HStore(1912): 5f2c97e00c770d329321cff2fbd6848c/A is initiating minor compaction (all files) 2023-04-25 02:55:55,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:55,048 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 5f2c97e00c770d329321cff2fbd6848c/A in TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. 2023-04-25 02:55:55,048 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37512 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-25 02:55:55,048 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/66eef2882bba40208d1637b2b78094c7, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/9e2fa0e41ba64e1782832107abb9d0ac, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/f7bc9d8bdc0b4454950c0f53439b3276] into tmpdir=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp, totalSize=41.4 K 2023-04-25 02:55:55,048 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HStore(1912): 5f2c97e00c770d329321cff2fbd6848c/B is initiating minor compaction (all files) 2023-04-25 02:55:55,048 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 5f2c97e00c770d329321cff2fbd6848c/B in TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. 2023-04-25 02:55:55,049 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/7ea43f735d6641d39ac3a4b33c99abf6, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/a8603cbb821b4b3ba43793987f38a2fc, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/662e556e867b4b4f92e9573ee399ab90] into tmpdir=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp, totalSize=36.6 K 2023-04-25 02:55:55,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:55,049 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting 7ea43f735d6641d39ac3a4b33c99abf6, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=424, earliestPutTs=1682391353236 2023-04-25 02:55:55,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:55,049 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting 66eef2882bba40208d1637b2b78094c7, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=424, earliestPutTs=1682391353236 2023-04-25 02:55:55,049 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting a8603cbb821b4b3ba43793987f38a2fc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=440, earliestPutTs=1682391353626 2023-04-25 02:55:55,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:55,049 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting 9e2fa0e41ba64e1782832107abb9d0ac, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=440, earliestPutTs=1682391353612 2023-04-25 02:55:55,050 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting 662e556e867b4b4f92e9573ee399ab90, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=460, earliestPutTs=1682391353675 2023-04-25 02:55:55,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:55,050 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting f7bc9d8bdc0b4454950c0f53439b3276, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=460, earliestPutTs=1682391353675 2023-04-25 02:55:55,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:55,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:55:55,050 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:55:55,050 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:55:55,051 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:55:55,051 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:55:55,051 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:55:55,051 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:55:55,053 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:55,053 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:55,053 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:55,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:55,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:55,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:55,053 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:55,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:55,053 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:55,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:55,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:55,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:55,055 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:55,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:55,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:55,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:55,056 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:55,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:55,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:55,058 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] throttle.PressureAwareThroughputController(145): 5f2c97e00c770d329321cff2fbd6848c#B#compaction#468 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 53.85 MB/second 2023-04-25 02:55:55,058 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] throttle.PressureAwareThroughputController(145): 5f2c97e00c770d329321cff2fbd6848c#A#compaction#469 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-25 02:55:55,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:55,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:55,062 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:55,062 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:55,062 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:55,062 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:55,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:55,062 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:55,062 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:55,062 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:55:55,062 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:55,062 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:55,062 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:55:55,062 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:55:55,062 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:55,062 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:55,062 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:55,063 DEBUG [Thread-2900] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:55:55,063 DEBUG [Thread-2900] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:55:55,064 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:55,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:55,075 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp/A/99353af3958249ae8526668303b5d352 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/99353af3958249ae8526668303b5d352 2023-04-25 02:55:55,080 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 5f2c97e00c770d329321cff2fbd6848c/A of 5f2c97e00c770d329321cff2fbd6848c into 99353af3958249ae8526668303b5d352(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 02:55:55,080 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5f2c97e00c770d329321cff2fbd6848c: 2023-04-25 02:55:55,080 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c., storeName=5f2c97e00c770d329321cff2fbd6848c/A, priority=13, startTime=1682391355047; duration=0sec 2023-04-25 02:55:55,080 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 02:55:55,080 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-25 02:55:55,081 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37512 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-25 02:55:55,081 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HStore(1912): 5f2c97e00c770d329321cff2fbd6848c/C is initiating minor compaction (all files) 2023-04-25 02:55:55,081 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 5f2c97e00c770d329321cff2fbd6848c/C in TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. 2023-04-25 02:55:55,082 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/1b75821bb43b4cf78fb3f5fcf43a3a7b, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/8d281d48a1a241a290c8e810179fd82c, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/9e7e07dea075439aabcb5d0ca4df3923] into tmpdir=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp, totalSize=36.6 K 2023-04-25 02:55:55,082 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting 1b75821bb43b4cf78fb3f5fcf43a3a7b, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=424, earliestPutTs=1682391353236 2023-04-25 02:55:55,082 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting 8d281d48a1a241a290c8e810179fd82c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=440, earliestPutTs=1682391353626 2023-04-25 02:55:55,083 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting 9e7e07dea075439aabcb5d0ca4df3923, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=460, earliestPutTs=1682391353675 2023-04-25 02:55:55,087 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] throttle.PressureAwareThroughputController(145): 5f2c97e00c770d329321cff2fbd6848c#C#compaction#470 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-25 02:55:55,097 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp/C/0fb3394889794077bc99b9a8df8e5482 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/0fb3394889794077bc99b9a8df8e5482 2023-04-25 02:55:55,102 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 5f2c97e00c770d329321cff2fbd6848c/C of 5f2c97e00c770d329321cff2fbd6848c into 0fb3394889794077bc99b9a8df8e5482(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 02:55:55,102 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5f2c97e00c770d329321cff2fbd6848c: 2023-04-25 02:55:55,102 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c., storeName=5f2c97e00c770d329321cff2fbd6848c/C, priority=13, startTime=1682391355047; duration=0sec 2023-04-25 02:55:55,102 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 02:55:55,163 DEBUG [Thread-2900] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:55:55,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:55:55,165 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:55:55,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:55:55,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:55:55,166 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:55:55,166 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:55,166 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:55:55,166 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:55:55,167 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:55,167 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:55,168 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:55,168 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:55,168 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:55,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:55,168 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:55,168 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:55:55,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:55,169 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:55,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:55:55,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:55,169 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:55:55,169 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:55:55,170 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:55:55,170 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:55:55,170 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:55,170 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:55:55,170 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:55:55,170 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:55:55,170 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. 2023-04-25 02:55:55,170 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. started... 2023-04-25 02:55:55,170 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5f2c97e00c770d329321cff2fbd6848c 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-25 02:55:55,170 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5f2c97e00c770d329321cff2fbd6848c, store=A 2023-04-25 02:55:55,170 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:55:55,170 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5f2c97e00c770d329321cff2fbd6848c, store=B 2023-04-25 02:55:55,170 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:55:55,170 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5f2c97e00c770d329321cff2fbd6848c, store=C 2023-04-25 02:55:55,171 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:55:55,182 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=477 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp/A/db52f1f90d44423db54e5238fd6df51f 2023-04-25 02:55:55,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(9158): Flush requested on 5f2c97e00c770d329321cff2fbd6848c 2023-04-25 02:55:55,345 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. as already flushing 2023-04-25 02:55:55,378 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:55,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59136 deadline: 1682391415374, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:55,379 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:55,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59162 deadline: 1682391415374, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:55,381 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:55,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59116 deadline: 1682391415375, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:55,478 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp/B/96fd73538d754ca7b56cfbe710cf7944 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/96fd73538d754ca7b56cfbe710cf7944 2023-04-25 02:55:55,484 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 5f2c97e00c770d329321cff2fbd6848c/B of 5f2c97e00c770d329321cff2fbd6848c into 96fd73538d754ca7b56cfbe710cf7944(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 02:55:55,484 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5f2c97e00c770d329321cff2fbd6848c: 2023-04-25 02:55:55,484 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c., storeName=5f2c97e00c770d329321cff2fbd6848c/B, priority=13, startTime=1682391355047; duration=0sec 2023-04-25 02:55:55,484 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 02:55:55,486 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:55,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59136 deadline: 1682391415480, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:55,486 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:55,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59162 deadline: 1682391415480, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:55,491 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:55,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59116 deadline: 1682391415483, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:55,593 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=477 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp/B/9f2365a7ca5f47da86303261d7e68dde 2023-04-25 02:55:55,690 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:55,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59136 deadline: 1682391415688, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:55,690 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:55,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59162 deadline: 1682391415689, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:55,697 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:55,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59116 deadline: 1682391415692, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:55,846 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:55,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59156 deadline: 1682391415843, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:55,874 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:55,875 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59114 deadline: 1682391415870, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:55,995 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:55,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59162 deadline: 1682391415992, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:55,996 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:55,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59136 deadline: 1682391415993, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:56,002 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:56,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59116 deadline: 1682391415999, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:56,004 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=477 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp/C/05cc74ac6aa64b3ebae3375a9fae09c6 2023-04-25 02:55:56,008 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp/A/db52f1f90d44423db54e5238fd6df51f as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/db52f1f90d44423db54e5238fd6df51f 2023-04-25 02:55:56,011 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/db52f1f90d44423db54e5238fd6df51f, entries=150, sequenceid=477, filesize=11.9 K 2023-04-25 02:55:56,012 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp/B/9f2365a7ca5f47da86303261d7e68dde as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/9f2365a7ca5f47da86303261d7e68dde 2023-04-25 02:55:56,015 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/9f2365a7ca5f47da86303261d7e68dde, entries=150, sequenceid=477, filesize=11.9 K 2023-04-25 02:55:56,016 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp/C/05cc74ac6aa64b3ebae3375a9fae09c6 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/05cc74ac6aa64b3ebae3375a9fae09c6 2023-04-25 02:55:56,028 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/05cc74ac6aa64b3ebae3375a9fae09c6, entries=150, sequenceid=477, filesize=11.9 K 2023-04-25 02:55:56,029 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for 5f2c97e00c770d329321cff2fbd6848c in 859ms, sequenceid=477, compaction requested=false 2023-04-25 02:55:56,029 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5f2c97e00c770d329321cff2fbd6848c: 2023-04-25 02:55:56,029 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. 2023-04-25 02:55:56,029 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:55:56,029 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:55:56,029 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:56,029 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:55:56,029 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:55:56,030 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:56,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:56,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:56,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:56,031 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:56,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:56,031 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:56,031 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:55:56,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:56,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:56,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:56,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:56,033 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:55:56,033 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5390bf44[Count = 0] remaining members to acquire global barrier 2023-04-25 02:55:56,033 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:55:56,033 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:56,034 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:56,034 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:56,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:56,034 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:55:56,034 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:55:56,034 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:55:56,039 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:55:56,039 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:56,039 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:55:56,039 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:56,039 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:55:56,039 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:55:56,039 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:55:56,039 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:55:56,040 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:55:56,040 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:55:56,040 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:55:56,041 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:56,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:56,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:56,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:56,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:56,041 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:56,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:56,042 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:56,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:56,042 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:56,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:56,042 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:56,043 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:56,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:56,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:56,043 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:56,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:56,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:56,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:56,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:56,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:56,045 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:56,045 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:56,045 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:56,045 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:56,045 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:56,045 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:56,045 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:56,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:56,045 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:55:56,046 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:55:56,045 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:56,045 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:56,046 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:56,045 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:56,046 DEBUG [Thread-2900] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:55:56,046 DEBUG [Thread-2900] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:55:56,046 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:56,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:56,046 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:55:56,146 DEBUG [Thread-2900] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:55:56,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:55:56,148 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:55:56,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:55:56,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:55:56,150 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:55:56,150 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:56,150 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:55:56,150 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:55:56,150 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:56,150 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:56,151 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:56,151 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:56,152 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:56,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:56,152 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:56,152 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:55:56,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:56,152 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:56,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:55:56,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:56,153 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:55:56,153 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:55:56,154 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:55:56,154 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:55:56,154 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:55:56,154 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:55:56,154 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:55:56,154 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:55:56,154 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. 2023-04-25 02:55:56,154 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. started... 2023-04-25 02:55:56,156 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5f2c97e00c770d329321cff2fbd6848c 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-25 02:55:56,156 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5f2c97e00c770d329321cff2fbd6848c, store=A 2023-04-25 02:55:56,156 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:55:56,156 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5f2c97e00c770d329321cff2fbd6848c, store=B 2023-04-25 02:55:56,156 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:55:56,156 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5f2c97e00c770d329321cff2fbd6848c, store=C 2023-04-25 02:55:56,156 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:55:56,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(9158): Flush requested on 5f2c97e00c770d329321cff2fbd6848c 2023-04-25 02:55:56,501 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. as already flushing 2023-04-25 02:55:56,527 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:56,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59116 deadline: 1682391416524, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:56,531 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:56,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59136 deadline: 1682391416526, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:56,531 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:56,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59162 deadline: 1682391416527, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:56,566 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=495 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp/A/dc0cebc2c6bd499192bc5b0c34c6d64b 2023-04-25 02:55:56,575 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=495 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp/B/4d4d8d8b49a642baa457ea8e0f6c5611 2023-04-25 02:55:56,634 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:56,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59116 deadline: 1682391416628, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:56,635 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:56,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59136 deadline: 1682391416632, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:56,635 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:56,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59162 deadline: 1682391416632, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:56,778 DEBUG [Thread-2905] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6ed183a4 to 127.0.0.1:62054 2023-04-25 02:55:56,778 DEBUG [Thread-2905] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-25 02:55:56,780 DEBUG [Thread-2903] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x549bef45 to 127.0.0.1:62054 2023-04-25 02:55:56,780 DEBUG [Thread-2903] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-25 02:55:56,781 DEBUG [Thread-2909] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3a9f3972 to 127.0.0.1:62054 2023-04-25 02:55:56,781 DEBUG [Thread-2909] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-25 02:55:56,783 DEBUG [Thread-2907] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x10a344e6 to 127.0.0.1:62054 2023-04-25 02:55:56,783 DEBUG [Thread-2907] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-25 02:55:56,785 DEBUG [Thread-2901] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0b475d8d to 127.0.0.1:62054 2023-04-25 02:55:56,785 DEBUG [Thread-2901] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-25 02:55:56,837 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:56,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59162 deadline: 1682391416836, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:56,837 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:56,837 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:55:56,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59116 deadline: 1682391416837, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:56,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59136 deadline: 1682391416837, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:56,984 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=495 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp/C/e4c2bbe78ead4de0a26a73e71dabf737 2023-04-25 02:55:56,987 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp/A/dc0cebc2c6bd499192bc5b0c34c6d64b as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/dc0cebc2c6bd499192bc5b0c34c6d64b 2023-04-25 02:55:56,990 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/dc0cebc2c6bd499192bc5b0c34c6d64b, entries=150, sequenceid=495, filesize=11.9 K 2023-04-25 02:55:56,991 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp/B/4d4d8d8b49a642baa457ea8e0f6c5611 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/4d4d8d8b49a642baa457ea8e0f6c5611 2023-04-25 02:55:56,994 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/4d4d8d8b49a642baa457ea8e0f6c5611, entries=150, sequenceid=495, filesize=11.9 K 2023-04-25 02:55:56,994 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp/C/e4c2bbe78ead4de0a26a73e71dabf737 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/e4c2bbe78ead4de0a26a73e71dabf737 2023-04-25 02:55:56,998 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/e4c2bbe78ead4de0a26a73e71dabf737, entries=150, sequenceid=495, filesize=11.9 K 2023-04-25 02:55:56,999 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for 5f2c97e00c770d329321cff2fbd6848c in 844ms, sequenceid=495, compaction requested=true 2023-04-25 02:55:56,999 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5f2c97e00c770d329321cff2fbd6848c: 2023-04-25 02:55:56,999 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. 2023-04-25 02:55:56,999 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:55:56,999 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:55:56,999 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:56,999 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:55:56,999 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:55:57,000 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:57,001 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:57,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:57,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:57,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:57,001 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:57,001 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:55:57,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:57,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:57,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:57,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:57,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:57,002 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:55:57,002 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@149c92ed[Count = 0] remaining members to acquire global barrier 2023-04-25 02:55:57,002 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:55:57,002 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:57,003 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:57,003 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:57,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:57,003 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:55:57,003 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:55:57,003 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:57,003 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:55:57,003 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:55:57,004 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:57,004 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:55:57,004 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:57,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:57,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:57,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:57,004 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:55:57,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:57,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:57,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:57,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:57,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:57,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:57,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:57,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:55:57,006 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:55:57,006 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:55:57,006 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:55:57,006 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:55:57,006 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:55:57,006 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:55:57,009 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:57,009 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:57,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:57,009 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:57,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:55:57,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:55:57,009 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:57,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:57,009 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:57,009 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:57,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:55:57,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:57,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:57,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:57,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:55:57,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:57,010 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:57,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:57,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:55:57,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:55:57,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:57,012 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:55:57,012 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:55:57,012 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:55:57,012 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:57,012 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:55:57,012 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:57,012 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:55:57,012 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:57,012 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:55:57,012 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:55:57,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:55:57,012 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:55:57,012 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:57,013 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:55:57,013 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:55:57,013 DEBUG [Thread-2900] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:55:57,013 DEBUG [Thread-2900] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:55:57,013 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:55:57,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:55:57,113 DEBUG [Thread-2900] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:55:57,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:55:57,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(9158): Flush requested on 5f2c97e00c770d329321cff2fbd6848c 2023-04-25 02:55:57,141 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 5f2c97e00c770d329321cff2fbd6848c 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-25 02:55:57,141 DEBUG [Thread-2894] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6f2b3c03 to 127.0.0.1:62054 2023-04-25 02:55:57,141 DEBUG [Thread-2894] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-25 02:55:57,141 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5f2c97e00c770d329321cff2fbd6848c, store=A 2023-04-25 02:55:57,141 DEBUG [Thread-2892] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5726fb8b to 127.0.0.1:62054 2023-04-25 02:55:57,141 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:55:57,141 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5f2c97e00c770d329321cff2fbd6848c, store=B 2023-04-25 02:55:57,141 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:55:57,141 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5f2c97e00c770d329321cff2fbd6848c, store=C 2023-04-25 02:55:57,141 DEBUG [Thread-2898] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x067ea274 to 127.0.0.1:62054 2023-04-25 02:55:57,141 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:55:57,141 DEBUG [Thread-2892] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-25 02:55:57,141 DEBUG [Thread-2898] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-25 02:55:57,548 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=513 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp/A/a10078154a434ab7b2a34e78edce337e 2023-04-25 02:55:57,861 DEBUG [Thread-2890] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x50002343 to 127.0.0.1:62054 2023-04-25 02:55:57,861 DEBUG [Thread-2890] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-25 02:55:57,882 DEBUG [Thread-2896] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x70e2be04 to 127.0.0.1:62054 2023-04-25 02:55:57,882 DEBUG [Thread-2896] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-25 02:55:57,882 INFO [Listener at localhost/41915] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-25 02:55:57,882 INFO [Listener at localhost/41915] hbase.AcidGuaranteesTestTool(394): wrote 92 2023-04-25 02:55:57,882 INFO [Listener at localhost/41915] hbase.AcidGuaranteesTestTool(394): wrote 69 2023-04-25 02:55:57,882 INFO [Listener at localhost/41915] hbase.AcidGuaranteesTestTool(394): wrote 75 2023-04-25 02:55:57,882 INFO [Listener at localhost/41915] hbase.AcidGuaranteesTestTool(394): wrote 89 2023-04-25 02:55:57,882 INFO [Listener at localhost/41915] hbase.AcidGuaranteesTestTool(394): wrote 75 2023-04-25 02:55:57,882 INFO [Listener at localhost/41915] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-25 02:55:57,882 INFO [Listener at localhost/41915] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-25 02:55:57,882 INFO [Listener at localhost/41915] hbase.AcidGuaranteesTestTool(402): scanned 1748 2023-04-25 02:55:57,882 INFO [Listener at localhost/41915] hbase.AcidGuaranteesTestTool(403): verified 5239 rows 2023-04-25 02:55:57,882 INFO [Listener at localhost/41915] hbase.AcidGuaranteesTestTool(402): scanned 1730 2023-04-25 02:55:57,882 INFO [Listener at localhost/41915] hbase.AcidGuaranteesTestTool(403): verified 5190 rows 2023-04-25 02:55:57,882 INFO [Listener at localhost/41915] hbase.AcidGuaranteesTestTool(402): scanned 1737 2023-04-25 02:55:57,882 INFO [Listener at localhost/41915] hbase.AcidGuaranteesTestTool(403): verified 5207 rows 2023-04-25 02:55:57,883 INFO [Listener at localhost/41915] hbase.AcidGuaranteesTestTool(402): scanned 1735 2023-04-25 02:55:57,883 INFO [Listener at localhost/41915] hbase.AcidGuaranteesTestTool(403): verified 5205 rows 2023-04-25 02:55:57,883 INFO [Listener at localhost/41915] hbase.AcidGuaranteesTestTool(402): scanned 1741 2023-04-25 02:55:57,883 INFO [Listener at localhost/41915] hbase.AcidGuaranteesTestTool(403): verified 5223 rows 2023-04-25 02:55:57,883 INFO [Listener at localhost/41915] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-25 02:55:57,883 DEBUG [Listener at localhost/41915] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x20ed9117 to 127.0.0.1:62054 2023-04-25 02:55:57,883 DEBUG [Listener at localhost/41915] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-25 02:55:57,886 INFO [Listener at localhost/41915] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-25 02:55:57,886 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-04-25 02:55:57,887 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure2.ProcedureExecutor(1029): Stored pid=50, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-25 02:55:57,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-25 02:55:57,890 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682391357890"}]},"ts":"1682391357890"} 2023-04-25 02:55:57,891 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-25 02:55:57,892 INFO [PEWorker-3] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-25 02:55:57,893 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=51, ppid=50, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=5f2c97e00c770d329321cff2fbd6848c, UNASSIGN}] 2023-04-25 02:55:57,894 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=51, ppid=50, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=5f2c97e00c770d329321cff2fbd6848c, UNASSIGN 2023-04-25 02:55:57,895 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=51 updating hbase:meta row=5f2c97e00c770d329321cff2fbd6848c, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:55:57,895 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682391357895"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682391357895"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682391357895"}]},"ts":"1682391357895"} 2023-04-25 02:55:57,896 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=52, ppid=51, state=RUNNABLE; CloseRegionProcedure 5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023}] 2023-04-25 02:55:57,957 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=513 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp/B/95c38976507b40e58f6b6dc8cb5c1bb7 2023-04-25 02:55:57,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-25 02:55:58,047 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 5f2c97e00c770d329321cff2fbd6848c 2023-04-25 02:55:58,047 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 5f2c97e00c770d329321cff2fbd6848c, disabling compactions & flushes 2023-04-25 02:55:58,047 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1858): waiting for 0 compactions & cache flush to complete for region TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. 2023-04-25 02:55:58,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-25 02:55:58,366 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=513 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp/C/78a15e5fd158405d8a9f5ece074f4387 2023-04-25 02:55:58,370 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp/A/a10078154a434ab7b2a34e78edce337e as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/a10078154a434ab7b2a34e78edce337e 2023-04-25 02:55:58,373 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/a10078154a434ab7b2a34e78edce337e, entries=150, sequenceid=513, filesize=11.9 K 2023-04-25 02:55:58,373 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp/B/95c38976507b40e58f6b6dc8cb5c1bb7 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/95c38976507b40e58f6b6dc8cb5c1bb7 2023-04-25 02:55:58,376 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/95c38976507b40e58f6b6dc8cb5c1bb7, entries=150, sequenceid=513, filesize=11.9 K 2023-04-25 02:55:58,377 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp/C/78a15e5fd158405d8a9f5ece074f4387 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/78a15e5fd158405d8a9f5ece074f4387 2023-04-25 02:55:58,380 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/78a15e5fd158405d8a9f5ece074f4387, entries=150, sequenceid=513, filesize=11.9 K 2023-04-25 02:55:58,380 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=13.42 KB/13740 for 5f2c97e00c770d329321cff2fbd6848c in 1240ms, sequenceid=513, compaction requested=true 2023-04-25 02:55:58,381 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 5f2c97e00c770d329321cff2fbd6848c: 2023-04-25 02:55:58,381 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. 2023-04-25 02:55:58,381 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. because compaction request was cancelled 2023-04-25 02:55:58,381 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 02:55:58,381 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. 2023-04-25 02:55:58,381 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 02:55:58,381 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. because compaction request was cancelled 2023-04-25 02:55:58,381 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. after waiting 0 ms 2023-04-25 02:55:58,381 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. because compaction request was cancelled 2023-04-25 02:55:58,381 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 02:55:58,381 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. 2023-04-25 02:55:58,381 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 5f2c97e00c770d329321cff2fbd6848c 3/3 column families, dataSize=13.42 KB heapSize=40.59 KB 2023-04-25 02:55:58,381 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5f2c97e00c770d329321cff2fbd6848c, store=A 2023-04-25 02:55:58,381 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:55:58,381 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5f2c97e00c770d329321cff2fbd6848c, store=B 2023-04-25 02:55:58,381 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:55:58,381 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5f2c97e00c770d329321cff2fbd6848c, store=C 2023-04-25 02:55:58,381 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:55:58,493 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-25 02:55:58,787 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=4.47 KB at sequenceid=518 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp/A/baaddf3894f34f93b616b1a89c75c022 2023-04-25 02:55:58,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-25 02:55:59,195 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=4.47 KB at sequenceid=518 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp/B/08a461dca3c7487c91cf28ff0ee1f120 2023-04-25 02:55:59,203 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=4.47 KB at sequenceid=518 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp/C/50c9d227729746a285123af40e0d60b0 2023-04-25 02:55:59,206 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp/A/baaddf3894f34f93b616b1a89c75c022 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/baaddf3894f34f93b616b1a89c75c022 2023-04-25 02:55:59,209 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/baaddf3894f34f93b616b1a89c75c022, entries=50, sequenceid=518, filesize=7.1 K 2023-04-25 02:55:59,209 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp/B/08a461dca3c7487c91cf28ff0ee1f120 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/08a461dca3c7487c91cf28ff0ee1f120 2023-04-25 02:55:59,212 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/08a461dca3c7487c91cf28ff0ee1f120, entries=50, sequenceid=518, filesize=7.1 K 2023-04-25 02:55:59,213 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/.tmp/C/50c9d227729746a285123af40e0d60b0 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/50c9d227729746a285123af40e0d60b0 2023-04-25 02:55:59,216 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/50c9d227729746a285123af40e0d60b0, entries=50, sequenceid=518, filesize=7.1 K 2023-04-25 02:55:59,216 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~13.42 KB/13740, heapSize ~40.55 KB/41520, currentSize=0 B/0 for 5f2c97e00c770d329321cff2fbd6848c in 835ms, sequenceid=518, compaction requested=true 2023-04-25 02:55:59,240 DEBUG [StoreCloser-TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/cac2fa2cedc34beabbd5ee13b997478c, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/23e4830258f74dbba9a73b11c9d5f826, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/329c77bde63b4a60bcaba2a4365c7d99, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/1713671593824abd8ab8ddb68b32e1b7, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/25bb38aa1ffa4896a8c2765283fe8746, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/4b83b1d55379429ba95b44494cb20aea, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/fbe6297e3cf8405cad592ff83411a75d, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/3aa6211c6eca46298db69d5be804516a, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/a80e49d868b64a3d9220fb8178b5d6a7, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/e112f6498cff4b439b8357b0c8ed59b0, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/52e807f0534b46299f8e5373f67266ba, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/287d09e1d924469b9e6a75a83fb1dd38, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/d117a9bf746a4d8595e45226929b8ab5, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/4fa064df1cdb48d48fc39f053e8f8992, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/d8598b45d90641cba7565004cbc1935d, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/2dd6823cb0444575b92b8c48203402dc, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/e960ee71989d4b839dc250da7ef43fc9, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/29dc053dd8cf4baca8fb1bcc48973535, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/d233cc3284d74a9986a80262b272c485, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/3ea22b53818540b59d1577f0406582ac, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/02a2d8e212ba4bac80d6c320de09f911, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/11d6d204318f4c4c87a006b0f44a5a17, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/c99f0d64b1574017ae53b1efa60850da, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/ef11df27c39f451e8247b0342327f248, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/bb2b08510e834f1abdcf36eb3b31e8ee, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/05c380d9cee547bc85d140c88cb8ec3a, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/f4efb953219346e786755552924893ea, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/0fad8006f82a41f8864f9dd51fe3abe0, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/66eef2882bba40208d1637b2b78094c7, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/9e2fa0e41ba64e1782832107abb9d0ac, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/f7bc9d8bdc0b4454950c0f53439b3276] to archive 2023-04-25 02:55:59,241 DEBUG [StoreCloser-TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-25 02:55:59,243 DEBUG [StoreCloser-TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/cac2fa2cedc34beabbd5ee13b997478c to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/cac2fa2cedc34beabbd5ee13b997478c 2023-04-25 02:55:59,244 DEBUG [StoreCloser-TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/23e4830258f74dbba9a73b11c9d5f826 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/23e4830258f74dbba9a73b11c9d5f826 2023-04-25 02:55:59,245 DEBUG [StoreCloser-TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/329c77bde63b4a60bcaba2a4365c7d99 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/329c77bde63b4a60bcaba2a4365c7d99 2023-04-25 02:55:59,245 DEBUG [StoreCloser-TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/1713671593824abd8ab8ddb68b32e1b7 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/1713671593824abd8ab8ddb68b32e1b7 2023-04-25 02:55:59,246 DEBUG [StoreCloser-TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/25bb38aa1ffa4896a8c2765283fe8746 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/25bb38aa1ffa4896a8c2765283fe8746 2023-04-25 02:55:59,247 DEBUG [StoreCloser-TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/4b83b1d55379429ba95b44494cb20aea to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/4b83b1d55379429ba95b44494cb20aea 2023-04-25 02:55:59,248 DEBUG [StoreCloser-TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/fbe6297e3cf8405cad592ff83411a75d to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/fbe6297e3cf8405cad592ff83411a75d 2023-04-25 02:55:59,249 DEBUG [StoreCloser-TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/3aa6211c6eca46298db69d5be804516a to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/3aa6211c6eca46298db69d5be804516a 2023-04-25 02:55:59,250 DEBUG [StoreCloser-TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/a80e49d868b64a3d9220fb8178b5d6a7 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/a80e49d868b64a3d9220fb8178b5d6a7 2023-04-25 02:55:59,251 DEBUG [StoreCloser-TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/e112f6498cff4b439b8357b0c8ed59b0 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/e112f6498cff4b439b8357b0c8ed59b0 2023-04-25 02:55:59,252 DEBUG [StoreCloser-TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/52e807f0534b46299f8e5373f67266ba to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/52e807f0534b46299f8e5373f67266ba 2023-04-25 02:55:59,252 DEBUG [StoreCloser-TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/287d09e1d924469b9e6a75a83fb1dd38 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/287d09e1d924469b9e6a75a83fb1dd38 2023-04-25 02:55:59,253 DEBUG [StoreCloser-TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/d117a9bf746a4d8595e45226929b8ab5 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/d117a9bf746a4d8595e45226929b8ab5 2023-04-25 02:55:59,254 DEBUG [StoreCloser-TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/4fa064df1cdb48d48fc39f053e8f8992 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/4fa064df1cdb48d48fc39f053e8f8992 2023-04-25 02:55:59,255 DEBUG [StoreCloser-TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/d8598b45d90641cba7565004cbc1935d to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/d8598b45d90641cba7565004cbc1935d 2023-04-25 02:55:59,256 DEBUG [StoreCloser-TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/2dd6823cb0444575b92b8c48203402dc to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/2dd6823cb0444575b92b8c48203402dc 2023-04-25 02:55:59,256 DEBUG [StoreCloser-TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/e960ee71989d4b839dc250da7ef43fc9 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/e960ee71989d4b839dc250da7ef43fc9 2023-04-25 02:55:59,257 DEBUG [StoreCloser-TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/29dc053dd8cf4baca8fb1bcc48973535 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/29dc053dd8cf4baca8fb1bcc48973535 2023-04-25 02:55:59,258 DEBUG [StoreCloser-TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/d233cc3284d74a9986a80262b272c485 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/d233cc3284d74a9986a80262b272c485 2023-04-25 02:55:59,259 DEBUG [StoreCloser-TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/3ea22b53818540b59d1577f0406582ac to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/3ea22b53818540b59d1577f0406582ac 2023-04-25 02:55:59,260 DEBUG [StoreCloser-TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/02a2d8e212ba4bac80d6c320de09f911 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/02a2d8e212ba4bac80d6c320de09f911 2023-04-25 02:55:59,260 DEBUG [StoreCloser-TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/11d6d204318f4c4c87a006b0f44a5a17 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/11d6d204318f4c4c87a006b0f44a5a17 2023-04-25 02:55:59,261 DEBUG [StoreCloser-TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/c99f0d64b1574017ae53b1efa60850da to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/c99f0d64b1574017ae53b1efa60850da 2023-04-25 02:55:59,262 DEBUG [StoreCloser-TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/ef11df27c39f451e8247b0342327f248 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/ef11df27c39f451e8247b0342327f248 2023-04-25 02:55:59,263 DEBUG [StoreCloser-TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/bb2b08510e834f1abdcf36eb3b31e8ee to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/bb2b08510e834f1abdcf36eb3b31e8ee 2023-04-25 02:55:59,264 DEBUG [StoreCloser-TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/05c380d9cee547bc85d140c88cb8ec3a to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/05c380d9cee547bc85d140c88cb8ec3a 2023-04-25 02:55:59,264 DEBUG [StoreCloser-TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/f4efb953219346e786755552924893ea to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/f4efb953219346e786755552924893ea 2023-04-25 02:55:59,265 DEBUG [StoreCloser-TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/0fad8006f82a41f8864f9dd51fe3abe0 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/0fad8006f82a41f8864f9dd51fe3abe0 2023-04-25 02:55:59,266 DEBUG [StoreCloser-TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/66eef2882bba40208d1637b2b78094c7 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/66eef2882bba40208d1637b2b78094c7 2023-04-25 02:55:59,267 DEBUG [StoreCloser-TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/9e2fa0e41ba64e1782832107abb9d0ac to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/9e2fa0e41ba64e1782832107abb9d0ac 2023-04-25 02:55:59,268 DEBUG [StoreCloser-TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/f7bc9d8bdc0b4454950c0f53439b3276 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/f7bc9d8bdc0b4454950c0f53439b3276 2023-04-25 02:55:59,270 DEBUG [StoreCloser-TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/1fc484b1f6d44b9aa4d98ea4c00115bc, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/2410eb188561495fa42936ef537ce3f9, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/1e555b288f224498a7ca167aaa30d2ad, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/c38c46225b7749fb978d49bdf27c4e26, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/041e58301c0042749998a15ae582afbb, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/6439852b16434808a0763062e981e154, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/902a85c141fe4d3683f8ac3de6c5e331, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/f503ad2c94fd41cbb1fc9a2e5502158d, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/925ee47c2491406cad64a1714667faa7, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/932e44e415b14d86aced9cb57cabdef5, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/1e20a7b405274c43bab50af4c413fc96, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/dbb2305ba9404b9aa60db115f822326c, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/c445318e69234d5887febfa5e5631384, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/447acc876db94cd584af1700e4ad7539, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/4218a8678fdf4ec2963e4bf68489839b, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/56d61f70a2d4416ab32bef05bd696a21, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/d26188e865fa4301becb61a3c08233a3, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/9a6aa4970f514d45a22dbd79b8707a39, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/0a4a8c56432f458a8eb20f3d505e7eec, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/5f55beaa53c541359b99496ee8f45d01, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/41af84bba07341a1856955d04d58cf9b, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/f681a8228f19424f8e43a7548ab53972, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/5ecbe96417ed42cf98ef2e791b9f9e80, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/6216c66b71c74e74ac217622b6c8b5eb, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/51b07b94467b421bbeaeba2d38e2afe1, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/e5a8d5db6d424e00b954889567b1e122, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/3c58268d9c214b2eb17a686707f58b4b, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/7ea43f735d6641d39ac3a4b33c99abf6, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/8343c71db4ae4de0967cd6a6977c3604, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/a8603cbb821b4b3ba43793987f38a2fc, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/662e556e867b4b4f92e9573ee399ab90] to archive 2023-04-25 02:55:59,271 DEBUG [StoreCloser-TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-25 02:55:59,272 DEBUG [StoreCloser-TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/1fc484b1f6d44b9aa4d98ea4c00115bc to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/1fc484b1f6d44b9aa4d98ea4c00115bc 2023-04-25 02:55:59,273 DEBUG [StoreCloser-TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/2410eb188561495fa42936ef537ce3f9 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/2410eb188561495fa42936ef537ce3f9 2023-04-25 02:55:59,274 DEBUG [StoreCloser-TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/1e555b288f224498a7ca167aaa30d2ad to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/1e555b288f224498a7ca167aaa30d2ad 2023-04-25 02:55:59,275 DEBUG [StoreCloser-TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/c38c46225b7749fb978d49bdf27c4e26 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/c38c46225b7749fb978d49bdf27c4e26 2023-04-25 02:55:59,276 DEBUG [StoreCloser-TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/041e58301c0042749998a15ae582afbb to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/041e58301c0042749998a15ae582afbb 2023-04-25 02:55:59,277 DEBUG [StoreCloser-TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/6439852b16434808a0763062e981e154 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/6439852b16434808a0763062e981e154 2023-04-25 02:55:59,278 DEBUG [StoreCloser-TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/902a85c141fe4d3683f8ac3de6c5e331 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/902a85c141fe4d3683f8ac3de6c5e331 2023-04-25 02:55:59,279 DEBUG [StoreCloser-TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/f503ad2c94fd41cbb1fc9a2e5502158d to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/f503ad2c94fd41cbb1fc9a2e5502158d 2023-04-25 02:55:59,280 DEBUG [StoreCloser-TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/925ee47c2491406cad64a1714667faa7 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/925ee47c2491406cad64a1714667faa7 2023-04-25 02:55:59,280 DEBUG [StoreCloser-TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/932e44e415b14d86aced9cb57cabdef5 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/932e44e415b14d86aced9cb57cabdef5 2023-04-25 02:55:59,281 DEBUG [StoreCloser-TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/1e20a7b405274c43bab50af4c413fc96 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/1e20a7b405274c43bab50af4c413fc96 2023-04-25 02:55:59,282 DEBUG [StoreCloser-TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/dbb2305ba9404b9aa60db115f822326c to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/dbb2305ba9404b9aa60db115f822326c 2023-04-25 02:55:59,283 DEBUG [StoreCloser-TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/c445318e69234d5887febfa5e5631384 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/c445318e69234d5887febfa5e5631384 2023-04-25 02:55:59,284 DEBUG [StoreCloser-TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/447acc876db94cd584af1700e4ad7539 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/447acc876db94cd584af1700e4ad7539 2023-04-25 02:55:59,284 DEBUG [StoreCloser-TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/4218a8678fdf4ec2963e4bf68489839b to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/4218a8678fdf4ec2963e4bf68489839b 2023-04-25 02:55:59,285 DEBUG [StoreCloser-TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/56d61f70a2d4416ab32bef05bd696a21 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/56d61f70a2d4416ab32bef05bd696a21 2023-04-25 02:55:59,286 DEBUG [StoreCloser-TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/d26188e865fa4301becb61a3c08233a3 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/d26188e865fa4301becb61a3c08233a3 2023-04-25 02:55:59,287 DEBUG [StoreCloser-TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/9a6aa4970f514d45a22dbd79b8707a39 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/9a6aa4970f514d45a22dbd79b8707a39 2023-04-25 02:55:59,288 DEBUG [StoreCloser-TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/0a4a8c56432f458a8eb20f3d505e7eec to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/0a4a8c56432f458a8eb20f3d505e7eec 2023-04-25 02:55:59,289 DEBUG [StoreCloser-TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/5f55beaa53c541359b99496ee8f45d01 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/5f55beaa53c541359b99496ee8f45d01 2023-04-25 02:55:59,290 DEBUG [StoreCloser-TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/41af84bba07341a1856955d04d58cf9b to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/41af84bba07341a1856955d04d58cf9b 2023-04-25 02:55:59,290 DEBUG [StoreCloser-TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/f681a8228f19424f8e43a7548ab53972 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/f681a8228f19424f8e43a7548ab53972 2023-04-25 02:55:59,291 DEBUG [StoreCloser-TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/5ecbe96417ed42cf98ef2e791b9f9e80 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/5ecbe96417ed42cf98ef2e791b9f9e80 2023-04-25 02:55:59,292 DEBUG [StoreCloser-TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/6216c66b71c74e74ac217622b6c8b5eb to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/6216c66b71c74e74ac217622b6c8b5eb 2023-04-25 02:55:59,293 DEBUG [StoreCloser-TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/51b07b94467b421bbeaeba2d38e2afe1 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/51b07b94467b421bbeaeba2d38e2afe1 2023-04-25 02:55:59,294 DEBUG [StoreCloser-TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/e5a8d5db6d424e00b954889567b1e122 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/e5a8d5db6d424e00b954889567b1e122 2023-04-25 02:55:59,294 DEBUG [StoreCloser-TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/3c58268d9c214b2eb17a686707f58b4b to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/3c58268d9c214b2eb17a686707f58b4b 2023-04-25 02:55:59,295 DEBUG [StoreCloser-TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/7ea43f735d6641d39ac3a4b33c99abf6 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/7ea43f735d6641d39ac3a4b33c99abf6 2023-04-25 02:55:59,296 DEBUG [StoreCloser-TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/8343c71db4ae4de0967cd6a6977c3604 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/8343c71db4ae4de0967cd6a6977c3604 2023-04-25 02:55:59,297 DEBUG [StoreCloser-TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/a8603cbb821b4b3ba43793987f38a2fc to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/a8603cbb821b4b3ba43793987f38a2fc 2023-04-25 02:55:59,298 DEBUG [StoreCloser-TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/662e556e867b4b4f92e9573ee399ab90 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/662e556e867b4b4f92e9573ee399ab90 2023-04-25 02:55:59,300 DEBUG [StoreCloser-TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/c39af132f78942f5a05423334d8ab1b6, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/a537c55d4a7b4b2990fa3405e4faa39b, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/05f9721c3c724ab0a2ac4f8dcd81f1a4, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/14a6f6a7152b44e48dc44f684cf6f6d6, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/bc783b0c9dc54965a05b5ae9c1b9a730, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/67b65249b4b04c09bdb0a8f5619d5d6e, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/8987c4e8b3f3427d8f4ea0faeb912a71, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/163ff9da8ebe4a6a8b338efd1173fdd4, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/ed9b1387563f4ed6a0e030fbcabd8c91, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/dc77376dad9d4835bdf1c9fa7fa07407, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/3928cbd922c14c2a8b0dd4c075905326, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/d53625636187492a9e6e3f4adabf357a, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/3379d58dc5514ea2a5beb85cd4a13556, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/664d7144148f4c78b163589760ecbb63, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/a00dc3705484472b9a6e4608a6b2c4f4, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/7f506986abab4b5089b3c8f0ddcc4400, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/696a9b716ea84776b15e96c7a7f9b67e, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/c27b7be3b60c421ca54880277c0f73ce, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/a925bc96190f4e5dbddaf31b5c850ca0, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/8b4d55c9537740638b13d13c4b4f4505, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/c0eb8b5035e94230963c2369fe6d38e9, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/80a67d4136f04b3a9560ad8185535640, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/89d7b0aeae6448e7a70466fca51e6eb3, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/b3275707397d42c1aed222e5ec36ff2c, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/482ed7ef431846318b4100732febde8a, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/361b6a9994fb47e8a3eca9a1973a0e6f, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/99ff5b002c1e42429cab7e3a978368ba, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/1b75821bb43b4cf78fb3f5fcf43a3a7b, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/4e6b3a559fc949b3bbee45b745e6fa56, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/8d281d48a1a241a290c8e810179fd82c, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/9e7e07dea075439aabcb5d0ca4df3923] to archive 2023-04-25 02:55:59,301 DEBUG [StoreCloser-TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-25 02:55:59,302 DEBUG [StoreCloser-TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/c39af132f78942f5a05423334d8ab1b6 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/c39af132f78942f5a05423334d8ab1b6 2023-04-25 02:55:59,303 DEBUG [StoreCloser-TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/a537c55d4a7b4b2990fa3405e4faa39b to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/a537c55d4a7b4b2990fa3405e4faa39b 2023-04-25 02:55:59,304 DEBUG [StoreCloser-TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/05f9721c3c724ab0a2ac4f8dcd81f1a4 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/05f9721c3c724ab0a2ac4f8dcd81f1a4 2023-04-25 02:55:59,305 DEBUG [StoreCloser-TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/14a6f6a7152b44e48dc44f684cf6f6d6 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/14a6f6a7152b44e48dc44f684cf6f6d6 2023-04-25 02:55:59,306 DEBUG [StoreCloser-TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/bc783b0c9dc54965a05b5ae9c1b9a730 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/bc783b0c9dc54965a05b5ae9c1b9a730 2023-04-25 02:55:59,307 DEBUG [StoreCloser-TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/67b65249b4b04c09bdb0a8f5619d5d6e to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/67b65249b4b04c09bdb0a8f5619d5d6e 2023-04-25 02:55:59,308 DEBUG [StoreCloser-TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/8987c4e8b3f3427d8f4ea0faeb912a71 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/8987c4e8b3f3427d8f4ea0faeb912a71 2023-04-25 02:55:59,309 DEBUG [StoreCloser-TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/163ff9da8ebe4a6a8b338efd1173fdd4 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/163ff9da8ebe4a6a8b338efd1173fdd4 2023-04-25 02:55:59,310 DEBUG [StoreCloser-TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/ed9b1387563f4ed6a0e030fbcabd8c91 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/ed9b1387563f4ed6a0e030fbcabd8c91 2023-04-25 02:55:59,310 DEBUG [StoreCloser-TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/dc77376dad9d4835bdf1c9fa7fa07407 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/dc77376dad9d4835bdf1c9fa7fa07407 2023-04-25 02:55:59,311 DEBUG [StoreCloser-TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/3928cbd922c14c2a8b0dd4c075905326 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/3928cbd922c14c2a8b0dd4c075905326 2023-04-25 02:55:59,312 DEBUG [StoreCloser-TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/d53625636187492a9e6e3f4adabf357a to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/d53625636187492a9e6e3f4adabf357a 2023-04-25 02:55:59,313 DEBUG [StoreCloser-TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/3379d58dc5514ea2a5beb85cd4a13556 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/3379d58dc5514ea2a5beb85cd4a13556 2023-04-25 02:55:59,313 DEBUG [StoreCloser-TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/664d7144148f4c78b163589760ecbb63 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/664d7144148f4c78b163589760ecbb63 2023-04-25 02:55:59,314 DEBUG [StoreCloser-TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/a00dc3705484472b9a6e4608a6b2c4f4 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/a00dc3705484472b9a6e4608a6b2c4f4 2023-04-25 02:55:59,315 DEBUG [StoreCloser-TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/7f506986abab4b5089b3c8f0ddcc4400 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/7f506986abab4b5089b3c8f0ddcc4400 2023-04-25 02:55:59,316 DEBUG [StoreCloser-TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/696a9b716ea84776b15e96c7a7f9b67e to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/696a9b716ea84776b15e96c7a7f9b67e 2023-04-25 02:55:59,317 DEBUG [StoreCloser-TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/c27b7be3b60c421ca54880277c0f73ce to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/c27b7be3b60c421ca54880277c0f73ce 2023-04-25 02:55:59,317 DEBUG [StoreCloser-TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/a925bc96190f4e5dbddaf31b5c850ca0 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/a925bc96190f4e5dbddaf31b5c850ca0 2023-04-25 02:55:59,318 DEBUG [StoreCloser-TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/8b4d55c9537740638b13d13c4b4f4505 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/8b4d55c9537740638b13d13c4b4f4505 2023-04-25 02:55:59,319 DEBUG [StoreCloser-TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/c0eb8b5035e94230963c2369fe6d38e9 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/c0eb8b5035e94230963c2369fe6d38e9 2023-04-25 02:55:59,320 DEBUG [StoreCloser-TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/80a67d4136f04b3a9560ad8185535640 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/80a67d4136f04b3a9560ad8185535640 2023-04-25 02:55:59,320 DEBUG [StoreCloser-TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/89d7b0aeae6448e7a70466fca51e6eb3 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/89d7b0aeae6448e7a70466fca51e6eb3 2023-04-25 02:55:59,321 DEBUG [StoreCloser-TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/b3275707397d42c1aed222e5ec36ff2c to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/b3275707397d42c1aed222e5ec36ff2c 2023-04-25 02:55:59,322 DEBUG [StoreCloser-TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/482ed7ef431846318b4100732febde8a to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/482ed7ef431846318b4100732febde8a 2023-04-25 02:55:59,323 DEBUG [StoreCloser-TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/361b6a9994fb47e8a3eca9a1973a0e6f to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/361b6a9994fb47e8a3eca9a1973a0e6f 2023-04-25 02:55:59,324 DEBUG [StoreCloser-TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/99ff5b002c1e42429cab7e3a978368ba to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/99ff5b002c1e42429cab7e3a978368ba 2023-04-25 02:55:59,324 DEBUG [StoreCloser-TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/1b75821bb43b4cf78fb3f5fcf43a3a7b to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/1b75821bb43b4cf78fb3f5fcf43a3a7b 2023-04-25 02:55:59,325 DEBUG [StoreCloser-TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/4e6b3a559fc949b3bbee45b745e6fa56 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/4e6b3a559fc949b3bbee45b745e6fa56 2023-04-25 02:55:59,326 DEBUG [StoreCloser-TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/8d281d48a1a241a290c8e810179fd82c to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/8d281d48a1a241a290c8e810179fd82c 2023-04-25 02:55:59,327 DEBUG [StoreCloser-TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/9e7e07dea075439aabcb5d0ca4df3923 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/9e7e07dea075439aabcb5d0ca4df3923 2023-04-25 02:55:59,331 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/recovered.edits/521.seqid, newMaxSeqId=521, maxSeqId=1 2023-04-25 02:55:59,333 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c. 2023-04-25 02:55:59,333 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 5f2c97e00c770d329321cff2fbd6848c: 2023-04-25 02:55:59,334 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 5f2c97e00c770d329321cff2fbd6848c 2023-04-25 02:55:59,334 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=51 updating hbase:meta row=5f2c97e00c770d329321cff2fbd6848c, regionState=CLOSED 2023-04-25 02:55:59,335 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682391359334"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682391359334"}]},"ts":"1682391359334"} 2023-04-25 02:55:59,336 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=52, resume processing ppid=51 2023-04-25 02:55:59,337 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=52, ppid=51, state=SUCCESS; CloseRegionProcedure 5f2c97e00c770d329321cff2fbd6848c, server=jenkins-hbase4.apache.org,46697,1682391220023 in 1.4390 sec 2023-04-25 02:55:59,337 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=51, resume processing ppid=50 2023-04-25 02:55:59,338 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=51, ppid=50, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=5f2c97e00c770d329321cff2fbd6848c, UNASSIGN in 1.4440 sec 2023-04-25 02:55:59,338 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682391359338"}]},"ts":"1682391359338"} 2023-04-25 02:55:59,339 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-25 02:55:59,341 INFO [PEWorker-3] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-25 02:55:59,342 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=50, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.4560 sec 2023-04-25 02:55:59,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-25 02:55:59,994 INFO [Listener at localhost/41915] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 50 completed 2023-04-25 02:55:59,995 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-04-25 02:55:59,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure2.ProcedureExecutor(1029): Stored pid=53, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-25 02:55:59,997 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=53, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-25 02:55:59,998 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=53, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-25 02:55:59,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1227): Checking to see if procedure is done pid=53 2023-04-25 02:56:00,002 DEBUG [HFileArchiver-11] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/.tmp/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c 2023-04-25 02:56:00,004 DEBUG [HFileArchiver-11] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/.tmp/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A, FileablePath, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/.tmp/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B, FileablePath, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/.tmp/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C, FileablePath, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/.tmp/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/recovered.edits] 2023-04-25 02:56:00,006 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/.tmp/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/99353af3958249ae8526668303b5d352 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/99353af3958249ae8526668303b5d352 2023-04-25 02:56:00,007 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/.tmp/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/a10078154a434ab7b2a34e78edce337e to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/a10078154a434ab7b2a34e78edce337e 2023-04-25 02:56:00,008 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/.tmp/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/baaddf3894f34f93b616b1a89c75c022 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/baaddf3894f34f93b616b1a89c75c022 2023-04-25 02:56:00,009 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/.tmp/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/db52f1f90d44423db54e5238fd6df51f to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/db52f1f90d44423db54e5238fd6df51f 2023-04-25 02:56:00,010 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/.tmp/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/dc0cebc2c6bd499192bc5b0c34c6d64b to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/A/dc0cebc2c6bd499192bc5b0c34c6d64b 2023-04-25 02:56:00,012 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/.tmp/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/08a461dca3c7487c91cf28ff0ee1f120 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/08a461dca3c7487c91cf28ff0ee1f120 2023-04-25 02:56:00,013 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/.tmp/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/4d4d8d8b49a642baa457ea8e0f6c5611 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/4d4d8d8b49a642baa457ea8e0f6c5611 2023-04-25 02:56:00,014 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/.tmp/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/95c38976507b40e58f6b6dc8cb5c1bb7 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/95c38976507b40e58f6b6dc8cb5c1bb7 2023-04-25 02:56:00,015 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/.tmp/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/96fd73538d754ca7b56cfbe710cf7944 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/96fd73538d754ca7b56cfbe710cf7944 2023-04-25 02:56:00,016 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/.tmp/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/9f2365a7ca5f47da86303261d7e68dde to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/B/9f2365a7ca5f47da86303261d7e68dde 2023-04-25 02:56:00,018 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/.tmp/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/05cc74ac6aa64b3ebae3375a9fae09c6 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/05cc74ac6aa64b3ebae3375a9fae09c6 2023-04-25 02:56:00,019 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/.tmp/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/0fb3394889794077bc99b9a8df8e5482 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/0fb3394889794077bc99b9a8df8e5482 2023-04-25 02:56:00,020 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/.tmp/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/50c9d227729746a285123af40e0d60b0 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/50c9d227729746a285123af40e0d60b0 2023-04-25 02:56:00,021 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/.tmp/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/78a15e5fd158405d8a9f5ece074f4387 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/78a15e5fd158405d8a9f5ece074f4387 2023-04-25 02:56:00,022 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/.tmp/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/e4c2bbe78ead4de0a26a73e71dabf737 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/C/e4c2bbe78ead4de0a26a73e71dabf737 2023-04-25 02:56:00,024 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/.tmp/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/recovered.edits/521.seqid to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c/recovered.edits/521.seqid 2023-04-25 02:56:00,024 DEBUG [HFileArchiver-11] backup.HFileArchiver(596): Deleted hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/.tmp/data/default/TestAcidGuarantees/5f2c97e00c770d329321cff2fbd6848c 2023-04-25 02:56:00,025 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-25 02:56:00,026 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=53, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-25 02:56:00,027 WARN [PEWorker-1] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-25 02:56:00,029 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-25 02:56:00,029 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=53, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-25 02:56:00,030 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-25 02:56:00,030 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1682391360030"}]},"ts":"9223372036854775807"} 2023-04-25 02:56:00,031 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-25 02:56:00,031 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 5f2c97e00c770d329321cff2fbd6848c, NAME => 'TestAcidGuarantees,,1682391334544.5f2c97e00c770d329321cff2fbd6848c.', STARTKEY => '', ENDKEY => ''}] 2023-04-25 02:56:00,031 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-25 02:56:00,031 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1682391360031"}]},"ts":"9223372036854775807"} 2023-04-25 02:56:00,032 INFO [PEWorker-1] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-25 02:56:00,034 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(130): Finished pid=53, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-25 02:56:00,035 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=53, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 39 msec 2023-04-25 02:56:00,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1227): Checking to see if procedure is done pid=53 2023-04-25 02:56:00,099 INFO [Listener at localhost/41915] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 53 completed 2023-04-25 02:56:00,112 INFO [Listener at localhost/41915] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testScanAtomicity Thread=229 (was 225) - Thread LEAK? -, OpenFileDescriptor=605 (was 528) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=469 (was 418) - SystemLoadAverage LEAK? -, ProcessCount=170 (was 170), AvailableMemoryMB=4515 (was 4191) - AvailableMemoryMB LEAK? - 2023-04-25 02:56:00,126 INFO [Listener at localhost/41915] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMobGetAtomicity Thread=229, OpenFileDescriptor=605, MaxFileDescriptor=60000, SystemLoadAverage=469, ProcessCount=170, AvailableMemoryMB=4515 2023-04-25 02:56:00,128 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-25 02:56:00,129 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-25 02:56:00,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure2.ProcedureExecutor(1029): Stored pid=54, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-25 02:56:00,131 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-25 02:56:00,131 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 54 2023-04-25 02:56:00,131 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-25 02:56:00,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-25 02:56:00,132 DEBUG [HFileArchiver-12] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/.tmp/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce 2023-04-25 02:56:00,133 DEBUG [HFileArchiver-12] backup.HFileArchiver(153): Directory hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/.tmp/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce empty. 2023-04-25 02:56:00,133 DEBUG [HFileArchiver-12] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/.tmp/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce 2023-04-25 02:56:00,133 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-25 02:56:00,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-25 02:56:00,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-25 02:56:00,540 DEBUG [PEWorker-5] util.FSTableDescriptors(570): Wrote into hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-25 02:56:00,541 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => d6e38fead90b2a328d380bf5577ff0ce, NAME => 'TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/.tmp 2023-04-25 02:56:00,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-25 02:56:00,947 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-25 02:56:00,947 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing d6e38fead90b2a328d380bf5577ff0ce, disabling compactions & flushes 2023-04-25 02:56:00,947 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. 2023-04-25 02:56:00,947 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. 2023-04-25 02:56:00,947 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. after waiting 0 ms 2023-04-25 02:56:00,947 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. 2023-04-25 02:56:00,947 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. 2023-04-25 02:56:00,947 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for d6e38fead90b2a328d380bf5577ff0ce: 2023-04-25 02:56:00,949 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-25 02:56:00,950 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682391360950"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682391360950"}]},"ts":"1682391360950"} 2023-04-25 02:56:00,951 INFO [PEWorker-5] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-25 02:56:00,952 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-25 02:56:00,952 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682391360952"}]},"ts":"1682391360952"} 2023-04-25 02:56:00,953 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-25 02:56:00,955 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=55, ppid=54, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=d6e38fead90b2a328d380bf5577ff0ce, ASSIGN}] 2023-04-25 02:56:00,957 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=55, ppid=54, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=d6e38fead90b2a328d380bf5577ff0ce, ASSIGN 2023-04-25 02:56:00,957 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=55, ppid=54, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=d6e38fead90b2a328d380bf5577ff0ce, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,46697,1682391220023; forceNewPlan=false, retain=false 2023-04-25 02:56:01,108 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=55 updating hbase:meta row=d6e38fead90b2a328d380bf5577ff0ce, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:01,109 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682391361108"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682391361108"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682391361108"}]},"ts":"1682391361108"} 2023-04-25 02:56:01,110 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=56, ppid=55, state=RUNNABLE; OpenRegionProcedure d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023}] 2023-04-25 02:56:01,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-25 02:56:01,264 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. 2023-04-25 02:56:01,264 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => d6e38fead90b2a328d380bf5577ff0ce, NAME => 'TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce.', STARTKEY => '', ENDKEY => ''} 2023-04-25 02:56:01,265 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees d6e38fead90b2a328d380bf5577ff0ce 2023-04-25 02:56:01,265 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-25 02:56:01,265 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for d6e38fead90b2a328d380bf5577ff0ce 2023-04-25 02:56:01,265 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for d6e38fead90b2a328d380bf5577ff0ce 2023-04-25 02:56:01,266 INFO [StoreOpener-d6e38fead90b2a328d380bf5577ff0ce-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region d6e38fead90b2a328d380bf5577ff0ce 2023-04-25 02:56:01,267 DEBUG [StoreOpener-d6e38fead90b2a328d380bf5577ff0ce-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/A 2023-04-25 02:56:01,267 DEBUG [StoreOpener-d6e38fead90b2a328d380bf5577ff0ce-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/A 2023-04-25 02:56:01,267 INFO [StoreOpener-d6e38fead90b2a328d380bf5577ff0ce-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-25 02:56:01,267 INFO [StoreOpener-d6e38fead90b2a328d380bf5577ff0ce-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region d6e38fead90b2a328d380bf5577ff0ce columnFamilyName A 2023-04-25 02:56:01,268 INFO [StoreOpener-d6e38fead90b2a328d380bf5577ff0ce-1] regionserver.HStore(310): Store=d6e38fead90b2a328d380bf5577ff0ce/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-25 02:56:01,268 INFO [StoreOpener-d6e38fead90b2a328d380bf5577ff0ce-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region d6e38fead90b2a328d380bf5577ff0ce 2023-04-25 02:56:01,269 DEBUG [StoreOpener-d6e38fead90b2a328d380bf5577ff0ce-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/B 2023-04-25 02:56:01,269 DEBUG [StoreOpener-d6e38fead90b2a328d380bf5577ff0ce-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/B 2023-04-25 02:56:01,269 INFO [StoreOpener-d6e38fead90b2a328d380bf5577ff0ce-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-25 02:56:01,269 INFO [StoreOpener-d6e38fead90b2a328d380bf5577ff0ce-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region d6e38fead90b2a328d380bf5577ff0ce columnFamilyName B 2023-04-25 02:56:01,270 INFO [StoreOpener-d6e38fead90b2a328d380bf5577ff0ce-1] regionserver.HStore(310): Store=d6e38fead90b2a328d380bf5577ff0ce/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-25 02:56:01,270 INFO [StoreOpener-d6e38fead90b2a328d380bf5577ff0ce-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region d6e38fead90b2a328d380bf5577ff0ce 2023-04-25 02:56:01,271 DEBUG [StoreOpener-d6e38fead90b2a328d380bf5577ff0ce-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/C 2023-04-25 02:56:01,271 DEBUG [StoreOpener-d6e38fead90b2a328d380bf5577ff0ce-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/C 2023-04-25 02:56:01,271 INFO [StoreOpener-d6e38fead90b2a328d380bf5577ff0ce-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-25 02:56:01,271 INFO [StoreOpener-d6e38fead90b2a328d380bf5577ff0ce-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region d6e38fead90b2a328d380bf5577ff0ce columnFamilyName C 2023-04-25 02:56:01,271 INFO [StoreOpener-d6e38fead90b2a328d380bf5577ff0ce-1] regionserver.HStore(310): Store=d6e38fead90b2a328d380bf5577ff0ce/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-25 02:56:01,272 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. 2023-04-25 02:56:01,272 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce 2023-04-25 02:56:01,272 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce 2023-04-25 02:56:01,274 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-25 02:56:01,275 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for d6e38fead90b2a328d380bf5577ff0ce 2023-04-25 02:56:01,276 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-25 02:56:01,277 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened d6e38fead90b2a328d380bf5577ff0ce; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=65489593, jitterRate=-0.024129018187522888}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-25 02:56:01,277 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for d6e38fead90b2a328d380bf5577ff0ce: 2023-04-25 02:56:01,277 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce., pid=56, masterSystemTime=1682391361261 2023-04-25 02:56:01,278 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. 2023-04-25 02:56:01,279 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. 2023-04-25 02:56:01,279 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=55 updating hbase:meta row=d6e38fead90b2a328d380bf5577ff0ce, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:01,279 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682391361279"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1682391361279"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1682391361279"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1682391361279"}]},"ts":"1682391361279"} 2023-04-25 02:56:01,281 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=56, resume processing ppid=55 2023-04-25 02:56:01,281 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=56, ppid=55, state=SUCCESS; OpenRegionProcedure d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 in 170 msec 2023-04-25 02:56:01,282 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=55, resume processing ppid=54 2023-04-25 02:56:01,282 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=55, ppid=54, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=d6e38fead90b2a328d380bf5577ff0ce, ASSIGN in 326 msec 2023-04-25 02:56:01,282 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-25 02:56:01,282 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682391361282"}]},"ts":"1682391361282"} 2023-04-25 02:56:01,283 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-25 02:56:01,287 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-25 02:56:01,287 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=54, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1580 sec 2023-04-25 02:56:02,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-25 02:56:02,236 INFO [Listener at localhost/41915] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 54 completed 2023-04-25 02:56:02,237 DEBUG [Listener at localhost/41915] zookeeper.ReadOnlyZKClient(139): Connect 0x59185ba7 to 127.0.0.1:62054 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-25 02:56:02,242 DEBUG [Listener at localhost/41915] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6af74c0e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-25 02:56:02,243 DEBUG [Listener at localhost/41915] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-25 02:56:02,244 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:39442, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-25 02:56:02,245 DEBUG [Listener at localhost/41915] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-25 02:56:02,247 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:56466, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-25 02:56:02,248 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-25 02:56:02,249 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.HMaster$12(2454): Client=jenkins//172.31.14.131 modify table TestAcidGuarantees from 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} to 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', MOB_THRESHOLD => '4', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-25 02:56:02,250 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure2.ProcedureExecutor(1029): Stored pid=57, state=RUNNABLE:MODIFY_TABLE_PREPARE; ModifyTableProcedure table=TestAcidGuarantees 2023-04-25 02:56:02,662 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-25 02:56:02,664 DEBUG [PEWorker-3] util.FSTableDescriptors(519): Deleted hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-25 02:56:02,664 INFO [PEWorker-3] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-25 02:56:02,666 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=58, ppid=57, state=RUNNABLE:REOPEN_TABLE_REGIONS_GET_REGIONS; ReopenTableRegionsProcedure table=TestAcidGuarantees}] 2023-04-25 02:56:02,668 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=59, ppid=58, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=d6e38fead90b2a328d380bf5577ff0ce, REOPEN/MOVE}] 2023-04-25 02:56:02,668 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=59, ppid=58, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=d6e38fead90b2a328d380bf5577ff0ce, REOPEN/MOVE 2023-04-25 02:56:02,669 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=d6e38fead90b2a328d380bf5577ff0ce, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:02,669 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682391362669"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682391362669"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682391362669"}]},"ts":"1682391362669"} 2023-04-25 02:56:02,670 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=60, ppid=59, state=RUNNABLE; CloseRegionProcedure d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023}] 2023-04-25 02:56:02,821 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close d6e38fead90b2a328d380bf5577ff0ce 2023-04-25 02:56:02,822 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing d6e38fead90b2a328d380bf5577ff0ce, disabling compactions & flushes 2023-04-25 02:56:02,822 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. 2023-04-25 02:56:02,822 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. 2023-04-25 02:56:02,822 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. after waiting 0 ms 2023-04-25 02:56:02,822 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. 2023-04-25 02:56:02,825 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-04-25 02:56:02,826 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. 2023-04-25 02:56:02,826 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for d6e38fead90b2a328d380bf5577ff0ce: 2023-04-25 02:56:02,826 WARN [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(3509): Not adding moved region record: d6e38fead90b2a328d380bf5577ff0ce to self. 2023-04-25 02:56:02,827 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed d6e38fead90b2a328d380bf5577ff0ce 2023-04-25 02:56:02,827 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=d6e38fead90b2a328d380bf5577ff0ce, regionState=CLOSED 2023-04-25 02:56:02,828 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682391362827"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682391362827"}]},"ts":"1682391362827"} 2023-04-25 02:56:02,829 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=60, resume processing ppid=59 2023-04-25 02:56:02,830 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=60, ppid=59, state=SUCCESS; CloseRegionProcedure d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 in 158 msec 2023-04-25 02:56:02,830 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=59, ppid=58, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=d6e38fead90b2a328d380bf5577ff0ce, REOPEN/MOVE; state=CLOSED, location=jenkins-hbase4.apache.org,46697,1682391220023; forceNewPlan=false, retain=true 2023-04-25 02:56:02,980 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=d6e38fead90b2a328d380bf5577ff0ce, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:02,981 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682391362980"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682391362980"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682391362980"}]},"ts":"1682391362980"} 2023-04-25 02:56:02,982 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=61, ppid=59, state=RUNNABLE; OpenRegionProcedure d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023}] 2023-04-25 02:56:03,136 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. 2023-04-25 02:56:03,136 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => d6e38fead90b2a328d380bf5577ff0ce, NAME => 'TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce.', STARTKEY => '', ENDKEY => ''} 2023-04-25 02:56:03,136 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees d6e38fead90b2a328d380bf5577ff0ce 2023-04-25 02:56:03,136 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-25 02:56:03,137 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for d6e38fead90b2a328d380bf5577ff0ce 2023-04-25 02:56:03,137 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for d6e38fead90b2a328d380bf5577ff0ce 2023-04-25 02:56:03,138 INFO [StoreOpener-d6e38fead90b2a328d380bf5577ff0ce-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region d6e38fead90b2a328d380bf5577ff0ce 2023-04-25 02:56:03,138 DEBUG [StoreOpener-d6e38fead90b2a328d380bf5577ff0ce-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/A 2023-04-25 02:56:03,138 DEBUG [StoreOpener-d6e38fead90b2a328d380bf5577ff0ce-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/A 2023-04-25 02:56:03,139 INFO [StoreOpener-d6e38fead90b2a328d380bf5577ff0ce-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-25 02:56:03,139 INFO [StoreOpener-d6e38fead90b2a328d380bf5577ff0ce-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region d6e38fead90b2a328d380bf5577ff0ce columnFamilyName A 2023-04-25 02:56:03,140 INFO [StoreOpener-d6e38fead90b2a328d380bf5577ff0ce-1] regionserver.HStore(310): Store=d6e38fead90b2a328d380bf5577ff0ce/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-25 02:56:03,140 INFO [StoreOpener-d6e38fead90b2a328d380bf5577ff0ce-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region d6e38fead90b2a328d380bf5577ff0ce 2023-04-25 02:56:03,141 DEBUG [StoreOpener-d6e38fead90b2a328d380bf5577ff0ce-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/B 2023-04-25 02:56:03,141 DEBUG [StoreOpener-d6e38fead90b2a328d380bf5577ff0ce-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/B 2023-04-25 02:56:03,141 INFO [StoreOpener-d6e38fead90b2a328d380bf5577ff0ce-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-25 02:56:03,141 INFO [StoreOpener-d6e38fead90b2a328d380bf5577ff0ce-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region d6e38fead90b2a328d380bf5577ff0ce columnFamilyName B 2023-04-25 02:56:03,141 INFO [StoreOpener-d6e38fead90b2a328d380bf5577ff0ce-1] regionserver.HStore(310): Store=d6e38fead90b2a328d380bf5577ff0ce/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-25 02:56:03,142 INFO [StoreOpener-d6e38fead90b2a328d380bf5577ff0ce-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region d6e38fead90b2a328d380bf5577ff0ce 2023-04-25 02:56:03,142 DEBUG [StoreOpener-d6e38fead90b2a328d380bf5577ff0ce-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/C 2023-04-25 02:56:03,142 DEBUG [StoreOpener-d6e38fead90b2a328d380bf5577ff0ce-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/C 2023-04-25 02:56:03,142 INFO [StoreOpener-d6e38fead90b2a328d380bf5577ff0ce-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-25 02:56:03,142 INFO [StoreOpener-d6e38fead90b2a328d380bf5577ff0ce-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region d6e38fead90b2a328d380bf5577ff0ce columnFamilyName C 2023-04-25 02:56:03,143 INFO [StoreOpener-d6e38fead90b2a328d380bf5577ff0ce-1] regionserver.HStore(310): Store=d6e38fead90b2a328d380bf5577ff0ce/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-25 02:56:03,143 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. 2023-04-25 02:56:03,143 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce 2023-04-25 02:56:03,144 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce 2023-04-25 02:56:03,145 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-25 02:56:03,146 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for d6e38fead90b2a328d380bf5577ff0ce 2023-04-25 02:56:03,147 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened d6e38fead90b2a328d380bf5577ff0ce; next sequenceid=5; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=74398403, jitterRate=0.10862259566783905}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-25 02:56:03,147 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for d6e38fead90b2a328d380bf5577ff0ce: 2023-04-25 02:56:03,147 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce., pid=61, masterSystemTime=1682391363133 2023-04-25 02:56:03,148 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. 2023-04-25 02:56:03,148 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. 2023-04-25 02:56:03,149 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=d6e38fead90b2a328d380bf5577ff0ce, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:03,149 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682391363149"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1682391363149"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1682391363149"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1682391363149"}]},"ts":"1682391363149"} 2023-04-25 02:56:03,151 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=61, resume processing ppid=59 2023-04-25 02:56:03,151 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=61, ppid=59, state=SUCCESS; OpenRegionProcedure d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 in 168 msec 2023-04-25 02:56:03,152 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=59, resume processing ppid=58 2023-04-25 02:56:03,152 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=59, ppid=58, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=d6e38fead90b2a328d380bf5577ff0ce, REOPEN/MOVE in 483 msec 2023-04-25 02:56:03,153 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=58, resume processing ppid=57 2023-04-25 02:56:03,154 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=58, ppid=57, state=SUCCESS; ReopenTableRegionsProcedure table=TestAcidGuarantees in 486 msec 2023-04-25 02:56:03,155 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=57, state=SUCCESS; ModifyTableProcedure table=TestAcidGuarantees in 905 msec 2023-04-25 02:56:03,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1227): Checking to see if procedure is done pid=57 2023-04-25 02:56:03,156 DEBUG [Listener at localhost/41915] zookeeper.ReadOnlyZKClient(139): Connect 0x062b3caa to 127.0.0.1:62054 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-25 02:56:03,161 DEBUG [Listener at localhost/41915] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1d759aab, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-25 02:56:03,162 DEBUG [Listener at localhost/41915] zookeeper.ReadOnlyZKClient(139): Connect 0x66b5b1f3 to 127.0.0.1:62054 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-25 02:56:03,167 DEBUG [Listener at localhost/41915] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@39241fe3, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-25 02:56:03,167 DEBUG [Listener at localhost/41915] zookeeper.ReadOnlyZKClient(139): Connect 0x544a4215 to 127.0.0.1:62054 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-25 02:56:03,171 DEBUG [Listener at localhost/41915] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1fbb5b46, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-25 02:56:03,171 DEBUG [Listener at localhost/41915] zookeeper.ReadOnlyZKClient(139): Connect 0x25baee3e to 127.0.0.1:62054 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-25 02:56:03,174 DEBUG [Listener at localhost/41915] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@23e65a76, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-25 02:56:03,174 DEBUG [Listener at localhost/41915] zookeeper.ReadOnlyZKClient(139): Connect 0x2c53bf96 to 127.0.0.1:62054 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-25 02:56:03,178 DEBUG [Listener at localhost/41915] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5e6a6f3c, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-25 02:56:03,178 DEBUG [Listener at localhost/41915] zookeeper.ReadOnlyZKClient(139): Connect 0x0a470a3d to 127.0.0.1:62054 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-25 02:56:03,182 DEBUG [Listener at localhost/41915] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7d997c6, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-25 02:56:03,183 DEBUG [Listener at localhost/41915] zookeeper.ReadOnlyZKClient(139): Connect 0x4bb72437 to 127.0.0.1:62054 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-25 02:56:03,187 DEBUG [Listener at localhost/41915] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7993d99e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-25 02:56:03,188 DEBUG [Listener at localhost/41915] zookeeper.ReadOnlyZKClient(139): Connect 0x3dca23c0 to 127.0.0.1:62054 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-25 02:56:03,195 DEBUG [Listener at localhost/41915] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4fae3ed2, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-25 02:56:03,196 DEBUG [Listener at localhost/41915] zookeeper.ReadOnlyZKClient(139): Connect 0x62d52321 to 127.0.0.1:62054 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-25 02:56:03,199 DEBUG [Listener at localhost/41915] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@eedd898, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-25 02:56:03,200 DEBUG [Listener at localhost/41915] zookeeper.ReadOnlyZKClient(139): Connect 0x2a8bf279 to 127.0.0.1:62054 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-25 02:56:03,206 DEBUG [Listener at localhost/41915] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@618c0022, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-25 02:56:03,214 DEBUG [hconnection-0x37c801c0-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-25 02:56:03,217 DEBUG [hconnection-0x9433678-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-25 02:56:03,220 DEBUG [hconnection-0x2e6ffcc8-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-25 02:56:03,220 DEBUG [hconnection-0x5fc58c1f-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-25 02:56:03,221 DEBUG [hconnection-0x1c443cd9-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-25 02:56:03,221 DEBUG [hconnection-0x39f9e47b-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-25 02:56:03,221 DEBUG [hconnection-0x40f47bc5-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-25 02:56:03,224 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:39482, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-25 02:56:03,224 DEBUG [hconnection-0xb99cc6d-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-25 02:56:03,226 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:39474, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-25 02:56:03,225 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:39458, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-25 02:56:03,226 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:39510, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-25 02:56:03,226 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:39494, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-25 02:56:03,227 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:56:03,229 DEBUG [hconnection-0x3f8eeb06-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-25 02:56:03,230 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:39536, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-25 02:56:03,233 DEBUG [hconnection-0x6a607dbc-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-25 02:56:03,234 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:39522, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-25 02:56:03,239 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:39500, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-25 02:56:03,239 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:39548, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-25 02:56:03,239 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:39526, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-25 02:56:03,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:56:03,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:56:03,244 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:56:03,244 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:56:03,244 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:56:03,244 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:56:03,245 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:03,245 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:03,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(9158): Flush requested on d6e38fead90b2a328d380bf5577ff0ce 2023-04-25 02:56:03,246 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing d6e38fead90b2a328d380bf5577ff0ce 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-25 02:56:03,247 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:03,247 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:56:03,247 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:56:03,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:56:03,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:03,248 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:03,248 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:56:03,249 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:03,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:56:03,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:03,249 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6e38fead90b2a328d380bf5577ff0ce, store=A 2023-04-25 02:56:03,249 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:56:03,249 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:56:03,249 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6e38fead90b2a328d380bf5577ff0ce, store=B 2023-04-25 02:56:03,249 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:56:03,249 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6e38fead90b2a328d380bf5577ff0ce, store=C 2023-04-25 02:56:03,249 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:56:03,249 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:56:03,249 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:56:03,249 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:56:03,250 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:56:03,250 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:56:03,250 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:56:03,250 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:56:03,250 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. 2023-04-25 02:56:03,250 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. started... 2023-04-25 02:56:03,250 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. as already flushing 2023-04-25 02:56:03,250 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. 2023-04-25 02:56:03,250 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:56:03,250 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:56:03,250 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:03,250 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:56:03,251 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:56:03,252 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:03,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:03,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:03,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:03,252 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:03,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:03,253 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:03,253 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:56:03,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:03,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:03,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:03,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:03,255 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:56:03,255 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@70f5b6a3[Count = 0] remaining members to acquire global barrier 2023-04-25 02:56:03,255 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:56:03,255 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:03,256 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:03,256 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:03,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:03,256 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:56:03,256 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:56:03,256 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:56:03,256 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:03,256 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:56:03,267 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:56:03,267 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:03,267 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:03,267 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:03,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39522 deadline: 1682391423263, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:03,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39482 deadline: 1682391423263, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:03,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:03,267 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:03,267 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:56:03,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39526 deadline: 1682391423264, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:03,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:03,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:03,267 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:56:03,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:03,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:03,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:03,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:03,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:03,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:03,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:03,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:56:03,269 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:56:03,269 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:56:03,270 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:56:03,270 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:56:03,270 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:56:03,270 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:56:03,270 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:03,270 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:03,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39536 deadline: 1682391423267, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:03,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39458 deadline: 1682391423267, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:03,272 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:03,272 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:03,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:03,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:03,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:03,272 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:03,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:03,272 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:56:03,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:03,272 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:56:03,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:56:03,272 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:03,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:03,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:03,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:03,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:03,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:03,274 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:03,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:03,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:03,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:03,276 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:56:03,276 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:56:03,276 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:56:03,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:56:03,277 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023042566508e177ca24eeeb8cb783530f8d7aa to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042566508e177ca24eeeb8cb783530f8d7aa 2023-04-25 02:56:03,277 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:03,277 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:03,277 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:03,277 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:03,277 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:03,277 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:03,277 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:03,277 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:56:03,277 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:56:03,277 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:56:03,277 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:56:03,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:56:03,277 DEBUG [Thread-3656] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:56:03,277 DEBUG [Thread-3656] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:56:03,278 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:56:03,369 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:03,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39482 deadline: 1682391423368, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:03,369 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:03,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39522 deadline: 1682391423368, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:03,370 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:03,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39526 deadline: 1682391423368, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:03,373 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:03,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39536 deadline: 1682391423373, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:03,376 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:03,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39458 deadline: 1682391423374, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:03,378 DEBUG [Thread-3656] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:56:03,378 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:56:03,380 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:56:03,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:56:03,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:56:03,381 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:56:03,381 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:56:03,381 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:56:03,381 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:56:03,382 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:03,382 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:03,383 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:56:03,383 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:56:03,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:56:03,383 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:03,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:03,383 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:03,383 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:56:03,384 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:03,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:56:03,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:03,385 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:56:03,385 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:56:03,385 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:56:03,385 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:56:03,385 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:56:03,385 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:56:03,385 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:56:03,385 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:56:03,385 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. 2023-04-25 02:56:03,386 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. started... 2023-04-25 02:56:03,386 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. as already flushing 2023-04-25 02:56:03,386 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. 2023-04-25 02:56:03,386 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:56:03,386 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:56:03,386 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:03,386 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:56:03,386 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:56:03,388 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:03,388 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:03,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:03,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:03,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:03,389 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:03,389 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:56:03,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:03,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:03,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:03,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:03,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:03,390 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:56:03,390 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1996b841[Count = 0] remaining members to acquire global barrier 2023-04-25 02:56:03,390 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:56:03,390 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:03,391 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:03,391 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:03,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:03,392 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:56:03,392 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:56:03,392 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:03,392 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:56:03,392 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:56:03,393 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:03,393 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:56:03,393 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:56:03,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:03,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:03,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:03,393 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:56:03,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:03,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:03,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:03,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:03,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:03,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:03,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:03,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:56:03,395 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:56:03,395 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:56:03,396 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:56:03,396 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:56:03,396 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:56:03,396 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:56:03,397 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:03,397 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:03,397 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:03,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:03,397 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:56:03,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:03,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:03,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:03,398 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:56:03,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:56:03,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:03,398 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:03,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:03,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:03,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:03,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:03,399 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:03,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:03,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:03,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:03,402 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:56:03,402 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:56:03,402 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:03,402 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:56:03,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:56:03,402 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:56:03,402 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:03,402 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:03,402 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:03,402 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:56:03,403 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:56:03,403 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:03,403 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:03,403 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:03,403 DEBUG [Thread-3656] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:56:03,403 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:56:03,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:56:03,403 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:56:03,403 DEBUG [Thread-3656] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:56:03,503 DEBUG [Thread-3656] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:56:03,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:56:03,506 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:56:03,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:56:03,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:56:03,507 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:56:03,507 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:56:03,507 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:56:03,507 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:56:03,508 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:03,508 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:03,509 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:03,509 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:56:03,509 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:56:03,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:56:03,509 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:03,509 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:56:03,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:03,510 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:03,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:56:03,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:03,510 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:56:03,510 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:56:03,511 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:56:03,511 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:56:03,511 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:56:03,511 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:56:03,511 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:56:03,511 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:56:03,511 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. 2023-04-25 02:56:03,511 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. started... 2023-04-25 02:56:03,511 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. as already flushing 2023-04-25 02:56:03,511 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. 2023-04-25 02:56:03,511 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:56:03,511 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:56:03,511 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:03,511 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:56:03,511 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:56:03,513 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:03,513 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:03,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:03,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:03,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:03,513 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:03,513 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:56:03,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:03,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:03,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:03,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:03,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:03,515 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:56:03,515 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4f2a4ea8[Count = 0] remaining members to acquire global barrier 2023-04-25 02:56:03,515 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:56:03,515 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:03,516 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:03,516 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:03,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:03,516 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:56:03,516 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:56:03,516 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:56:03,516 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:03,516 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:56:03,519 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:03,519 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:56:03,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:03,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:03,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:03,519 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:56:03,519 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:56:03,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:03,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:03,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:03,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:03,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:03,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:03,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:03,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:56:03,522 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:56:03,522 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:56:03,522 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:56:03,522 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:56:03,522 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:56:03,522 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:56:03,523 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:03,523 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:56:03,523 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:03,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:03,524 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:03,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:03,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:03,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:03,524 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:56:03,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:56:03,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:03,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:03,524 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:03,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:03,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:03,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:03,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:03,526 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:03,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:03,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:03,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:03,528 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:03,528 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:03,528 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:03,528 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:56:03,528 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:56:03,529 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:56:03,528 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:03,529 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:03,529 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:03,529 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:03,529 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:56:03,529 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:56:03,529 DEBUG [Thread-3656] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:56:03,529 DEBUG [Thread-3656] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:56:03,529 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:56:03,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:56:03,529 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:56:03,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:56:03,571 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:03,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39522 deadline: 1682391423571, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:03,572 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:03,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39482 deadline: 1682391423571, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:03,574 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:03,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39526 deadline: 1682391423572, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:03,577 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:03,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39536 deadline: 1682391423575, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:03,579 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:03,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39458 deadline: 1682391423578, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:03,629 DEBUG [Thread-3656] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:56:03,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:56:03,631 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:56:03,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:56:03,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:56:03,633 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:56:03,633 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:56:03,633 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:56:03,633 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:56:03,633 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:03,633 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:03,635 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:56:03,635 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:03,635 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:56:03,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:56:03,635 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:03,635 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:56:03,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:03,635 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:03,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:56:03,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:03,636 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:56:03,636 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:56:03,636 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:56:03,636 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:56:03,636 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:56:03,636 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:56:03,637 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:56:03,637 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:56:03,637 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. 2023-04-25 02:56:03,637 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. started... 2023-04-25 02:56:03,637 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. as already flushing 2023-04-25 02:56:03,637 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. 2023-04-25 02:56:03,637 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:56:03,637 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:56:03,637 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:03,637 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:56:03,637 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:56:03,643 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:03,643 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:03,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:03,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:03,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:03,643 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:03,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:03,643 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:56:03,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:03,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:03,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:03,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:03,645 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:56:03,645 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3f51a65a[Count = 0] remaining members to acquire global barrier 2023-04-25 02:56:03,645 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:56:03,645 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:03,646 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:03,646 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:03,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:03,646 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:56:03,646 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:56:03,646 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:56:03,646 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:03,646 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:56:03,647 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:56:03,647 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:56:03,647 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:56:03,647 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:03,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:03,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:03,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:03,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:03,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:03,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:03,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:03,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:03,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:03,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:03,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:56:03,650 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:56:03,650 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:56:03,650 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:56:03,650 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:56:03,650 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:56:03,650 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:56:03,655 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:03,655 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:03,655 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:03,655 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:56:03,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:03,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:03,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:03,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:03,656 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:56:03,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:56:03,656 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:03,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:03,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:03,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:03,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:03,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:03,657 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:03,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:03,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:03,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:03,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:03,660 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:03,660 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:03,660 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:03,660 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:56:03,660 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:56:03,660 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:56:03,660 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:56:03,660 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:56:03,660 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:03,660 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:56:03,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:56:03,660 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:03,660 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:56:03,661 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:03,661 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:03,661 DEBUG [Thread-3656] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:56:03,661 DEBUG [Thread-3656] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:56:03,661 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:56:03,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:56:03,690 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=16, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/.tmp/A/dfb20049ffd144eca1334779fe1d1e28 2023-04-25 02:56:03,705 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=16 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/.tmp/B/45c27d721ee4419aaf06aae8aa2eec83 2023-04-25 02:56:03,720 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=16 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/.tmp/C/24dc7dbdd2dc4078bb27b4eb8de7f60f 2023-04-25 02:56:03,726 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/.tmp/A/dfb20049ffd144eca1334779fe1d1e28 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/A/dfb20049ffd144eca1334779fe1d1e28 2023-04-25 02:56:03,730 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/A/dfb20049ffd144eca1334779fe1d1e28, entries=150, sequenceid=16, filesize=25.1 K 2023-04-25 02:56:03,731 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/.tmp/B/45c27d721ee4419aaf06aae8aa2eec83 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/B/45c27d721ee4419aaf06aae8aa2eec83 2023-04-25 02:56:03,734 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/B/45c27d721ee4419aaf06aae8aa2eec83, entries=150, sequenceid=16, filesize=11.6 K 2023-04-25 02:56:03,735 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/.tmp/C/24dc7dbdd2dc4078bb27b4eb8de7f60f as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/C/24dc7dbdd2dc4078bb27b4eb8de7f60f 2023-04-25 02:56:03,742 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/C/24dc7dbdd2dc4078bb27b4eb8de7f60f, entries=150, sequenceid=16, filesize=11.6 K 2023-04-25 02:56:03,743 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for d6e38fead90b2a328d380bf5577ff0ce in 497ms, sequenceid=16, compaction requested=false 2023-04-25 02:56:03,743 DEBUG [MemStoreFlusher.0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'TestAcidGuarantees' 2023-04-25 02:56:03,744 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for d6e38fead90b2a328d380bf5577ff0ce: 2023-04-25 02:56:03,761 DEBUG [Thread-3656] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:56:03,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:56:03,763 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:56:03,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:56:03,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:56:03,764 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:56:03,764 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:56:03,764 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:56:03,764 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:56:03,765 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:03,765 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:03,766 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:56:03,766 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:03,766 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:56:03,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:56:03,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:03,767 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:03,767 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:56:03,767 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:03,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:56:03,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:03,767 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:56:03,768 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:56:03,768 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:56:03,768 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:56:03,768 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:56:03,768 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:56:03,768 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:56:03,768 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:56:03,768 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. 2023-04-25 02:56:03,768 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. started... 2023-04-25 02:56:03,768 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d6e38fead90b2a328d380bf5577ff0ce 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-25 02:56:03,768 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6e38fead90b2a328d380bf5577ff0ce, store=A 2023-04-25 02:56:03,769 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:56:03,769 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6e38fead90b2a328d380bf5577ff0ce, store=B 2023-04-25 02:56:03,769 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:56:03,769 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6e38fead90b2a328d380bf5577ff0ce, store=C 2023-04-25 02:56:03,769 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:56:03,784 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304253b9c7bdc7b054d708a558d8f84b065cf to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304253b9c7bdc7b054d708a558d8f84b065cf 2023-04-25 02:56:03,803 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=36, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/.tmp/A/969bc1313d61424db21ef2e2fcf5c054 2023-04-25 02:56:03,875 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(9158): Flush requested on d6e38fead90b2a328d380bf5577ff0ce 2023-04-25 02:56:03,875 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. as already flushing 2023-04-25 02:56:03,881 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:03,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39482 deadline: 1682391423880, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:03,883 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:03,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39536 deadline: 1682391423881, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:03,883 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:03,883 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:03,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39522 deadline: 1682391423881, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:03,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39458 deadline: 1682391423881, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:03,884 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:03,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39526 deadline: 1682391423882, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:03,983 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:03,984 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39482 deadline: 1682391423983, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:03,986 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:03,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39536 deadline: 1682391423984, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:03,986 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:03,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39522 deadline: 1682391423984, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:03,986 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:03,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39526 deadline: 1682391423985, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:04,186 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:04,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39482 deadline: 1682391424185, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:04,188 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:04,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39536 deadline: 1682391424187, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:04,189 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:04,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39522 deadline: 1682391424187, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:04,189 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:04,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39526 deadline: 1682391424188, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:04,223 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=36 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/.tmp/B/0d44dc3aecf34e1595d7ef4216834a8f 2023-04-25 02:56:04,232 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=36 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/.tmp/C/5bec0d8e724d4b79a35ecf5d0a9c9b58 2023-04-25 02:56:04,237 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/.tmp/A/969bc1313d61424db21ef2e2fcf5c054 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/A/969bc1313d61424db21ef2e2fcf5c054 2023-04-25 02:56:04,241 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/A/969bc1313d61424db21ef2e2fcf5c054, entries=150, sequenceid=36, filesize=25.1 K 2023-04-25 02:56:04,241 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/.tmp/B/0d44dc3aecf34e1595d7ef4216834a8f as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/B/0d44dc3aecf34e1595d7ef4216834a8f 2023-04-25 02:56:04,245 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/B/0d44dc3aecf34e1595d7ef4216834a8f, entries=150, sequenceid=36, filesize=11.6 K 2023-04-25 02:56:04,248 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/.tmp/C/5bec0d8e724d4b79a35ecf5d0a9c9b58 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/C/5bec0d8e724d4b79a35ecf5d0a9c9b58 2023-04-25 02:56:04,256 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/C/5bec0d8e724d4b79a35ecf5d0a9c9b58, entries=150, sequenceid=36, filesize=11.6 K 2023-04-25 02:56:04,257 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for d6e38fead90b2a328d380bf5577ff0ce in 489ms, sequenceid=36, compaction requested=false 2023-04-25 02:56:04,258 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d6e38fead90b2a328d380bf5577ff0ce: 2023-04-25 02:56:04,258 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. 2023-04-25 02:56:04,258 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:56:04,258 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:56:04,258 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:04,258 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:56:04,258 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:56:04,259 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:04,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:04,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:04,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:04,260 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:04,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:04,260 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:04,260 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:56:04,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:04,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:04,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:04,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:04,261 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:56:04,262 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@624c5fbd[Count = 0] remaining members to acquire global barrier 2023-04-25 02:56:04,262 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:56:04,262 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:04,264 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:04,264 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:04,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:04,264 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:56:04,264 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:56:04,264 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:56:04,265 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:04,265 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:56:04,266 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:04,266 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:56:04,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:04,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:04,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:04,266 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:56:04,266 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:56:04,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:04,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:04,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:04,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:04,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:04,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:04,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:04,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:56:04,268 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:56:04,268 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:56:04,268 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:56:04,268 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:56:04,269 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:56:04,269 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:56:04,270 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:04,270 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:56:04,270 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:04,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:04,270 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:04,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:04,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:04,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:04,271 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:56:04,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:56:04,271 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:04,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:04,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:04,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:04,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:04,272 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:04,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:04,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:04,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:04,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:04,274 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:04,274 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:04,274 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:04,274 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:04,274 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:04,274 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:56:04,274 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:56:04,274 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:56:04,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:56:04,275 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:04,275 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:04,275 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:56:04,275 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:56:04,275 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:56:04,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:56:04,275 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:56:04,275 DEBUG [Thread-3656] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:56:04,275 DEBUG [Thread-3656] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:56:04,275 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:56:04,375 DEBUG [Thread-3656] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:56:04,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:56:04,377 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:56:04,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:56:04,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:56:04,379 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:56:04,379 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:56:04,379 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:56:04,379 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:56:04,379 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:04,379 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:04,381 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:56:04,381 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:04,381 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:56:04,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:56:04,382 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:04,382 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:56:04,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:04,382 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:04,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:56:04,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:04,383 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:56:04,383 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:56:04,383 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:56:04,383 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:56:04,383 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:56:04,383 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:56:04,383 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:56:04,383 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:56:04,383 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. 2023-04-25 02:56:04,383 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. started... 2023-04-25 02:56:04,383 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d6e38fead90b2a328d380bf5577ff0ce 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-25 02:56:04,384 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6e38fead90b2a328d380bf5577ff0ce, store=A 2023-04-25 02:56:04,384 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:56:04,384 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6e38fead90b2a328d380bf5577ff0ce, store=B 2023-04-25 02:56:04,384 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:56:04,384 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6e38fead90b2a328d380bf5577ff0ce, store=C 2023-04-25 02:56:04,384 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:56:04,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(9158): Flush requested on d6e38fead90b2a328d380bf5577ff0ce 2023-04-25 02:56:04,387 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. as already flushing 2023-04-25 02:56:04,405 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304251ba93b2f07e846a4aba7c533631c217a to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304251ba93b2f07e846a4aba7c533631c217a 2023-04-25 02:56:04,437 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:04,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39458 deadline: 1682391424435, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:04,488 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:04,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39482 deadline: 1682391424488, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:04,491 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:04,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39522 deadline: 1682391424490, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:04,491 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:04,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39526 deadline: 1682391424490, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:04,492 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:04,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39536 deadline: 1682391424490, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:04,540 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:04,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39458 deadline: 1682391424538, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:04,744 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:04,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39458 deadline: 1682391424742, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:04,810 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=48, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/.tmp/A/c269266e65534535b43d63130a94f865 2023-04-25 02:56:04,861 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-25 02:56:04,993 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:04,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39482 deadline: 1682391424992, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:04,995 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:04,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39536 deadline: 1682391424993, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:04,995 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:04,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39526 deadline: 1682391424994, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:04,997 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:04,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39522 deadline: 1682391424996, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:05,049 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:05,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39458 deadline: 1682391425046, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:05,221 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=48 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/.tmp/B/777035c54c8f48b4b5899b5d576af640 2023-04-25 02:56:05,553 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:05,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39458 deadline: 1682391425551, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:05,634 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=48 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/.tmp/C/3d3f3b9b61a84f8db4adefcfef0d6cad 2023-04-25 02:56:05,638 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/.tmp/A/c269266e65534535b43d63130a94f865 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/A/c269266e65534535b43d63130a94f865 2023-04-25 02:56:05,642 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/A/c269266e65534535b43d63130a94f865, entries=150, sequenceid=48, filesize=25.1 K 2023-04-25 02:56:05,643 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/.tmp/B/777035c54c8f48b4b5899b5d576af640 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/B/777035c54c8f48b4b5899b5d576af640 2023-04-25 02:56:05,650 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/B/777035c54c8f48b4b5899b5d576af640, entries=150, sequenceid=48, filesize=11.6 K 2023-04-25 02:56:05,651 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/.tmp/C/3d3f3b9b61a84f8db4adefcfef0d6cad as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/C/3d3f3b9b61a84f8db4adefcfef0d6cad 2023-04-25 02:56:05,654 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/C/3d3f3b9b61a84f8db4adefcfef0d6cad, entries=150, sequenceid=48, filesize=11.6 K 2023-04-25 02:56:05,655 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for d6e38fead90b2a328d380bf5577ff0ce in 1272ms, sequenceid=48, compaction requested=true 2023-04-25 02:56:05,655 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d6e38fead90b2a328d380bf5577ff0ce: 2023-04-25 02:56:05,655 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. 2023-04-25 02:56:05,655 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:56:05,655 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:56:05,655 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:05,655 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:56:05,655 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:56:05,657 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:05,657 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:05,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:05,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:05,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:05,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:05,657 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:05,657 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:56:05,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:05,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:05,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:05,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:05,659 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:56:05,659 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4eb4a705[Count = 0] remaining members to acquire global barrier 2023-04-25 02:56:05,659 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:56:05,659 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:05,660 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:05,660 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:05,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:05,660 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:56:05,660 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:56:05,660 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:56:05,660 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:05,660 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:56:05,661 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:56:05,662 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:56:05,662 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:05,662 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:56:05,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:05,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:05,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:05,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:05,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:05,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:05,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:05,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:05,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:05,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:05,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:56:05,664 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:56:05,664 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:56:05,665 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:56:05,665 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:56:05,665 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:56:05,665 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:56:05,667 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:05,667 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:05,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:05,667 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:05,667 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:56:05,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:05,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:05,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:05,668 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:56:05,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:56:05,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:05,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:05,668 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:05,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:05,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:05,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:05,670 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:05,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:05,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:05,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:05,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:05,673 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:56:05,673 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:56:05,673 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:56:05,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:56:05,673 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:05,673 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:56:05,673 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:56:05,673 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:56:05,673 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:56:05,673 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:56:05,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:56:05,673 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:05,673 DEBUG [Thread-3656] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:56:05,673 DEBUG [Thread-3656] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:56:05,673 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:05,673 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:05,673 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:05,673 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:05,673 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:05,773 DEBUG [Thread-3656] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:56:05,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:56:05,776 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:56:05,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:56:05,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:56:05,777 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:56:05,777 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:56:05,777 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:56:05,777 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:56:05,778 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:05,778 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:05,779 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:05,779 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:56:05,779 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:56:05,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:56:05,779 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:05,779 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:56:05,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:05,780 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:05,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:56:05,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:05,780 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:56:05,781 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:56:05,781 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:56:05,781 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:56:05,781 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:56:05,781 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:56:05,781 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:56:05,781 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:56:05,781 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. 2023-04-25 02:56:05,781 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. started... 2023-04-25 02:56:05,781 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d6e38fead90b2a328d380bf5577ff0ce 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-25 02:56:05,782 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6e38fead90b2a328d380bf5577ff0ce, store=A 2023-04-25 02:56:05,782 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:56:05,782 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6e38fead90b2a328d380bf5577ff0ce, store=B 2023-04-25 02:56:05,782 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:56:05,782 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6e38fead90b2a328d380bf5577ff0ce, store=C 2023-04-25 02:56:05,782 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:56:05,797 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304250579521144a74c9c88f374a2be1fe11f to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304250579521144a74c9c88f374a2be1fe11f 2023-04-25 02:56:05,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(9158): Flush requested on d6e38fead90b2a328d380bf5577ff0ce 2023-04-25 02:56:05,999 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. as already flushing 2023-04-25 02:56:06,010 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:06,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39536 deadline: 1682391426005, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:06,010 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:06,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39522 deadline: 1682391426006, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:06,010 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:06,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39482 deadline: 1682391426007, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:06,012 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:06,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39526 deadline: 1682391426010, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:06,112 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:06,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39536 deadline: 1682391426111, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:06,112 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:06,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39522 deadline: 1682391426111, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:06,113 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:06,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39482 deadline: 1682391426111, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:06,115 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:06,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39526 deadline: 1682391426113, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:06,206 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=68, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/.tmp/A/6e508d77291c4970bbe3d5ea0c612ece 2023-04-25 02:56:06,215 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=68 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/.tmp/B/13aa7915d1df47d6909e42a1c60402a3 2023-04-25 02:56:06,224 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=68 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/.tmp/C/4b534914c5064cea8af561d5965651c6 2023-04-25 02:56:06,229 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/.tmp/A/6e508d77291c4970bbe3d5ea0c612ece as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/A/6e508d77291c4970bbe3d5ea0c612ece 2023-04-25 02:56:06,234 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/A/6e508d77291c4970bbe3d5ea0c612ece, entries=150, sequenceid=68, filesize=25.1 K 2023-04-25 02:56:06,234 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/.tmp/B/13aa7915d1df47d6909e42a1c60402a3 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/B/13aa7915d1df47d6909e42a1c60402a3 2023-04-25 02:56:06,242 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/B/13aa7915d1df47d6909e42a1c60402a3, entries=150, sequenceid=68, filesize=11.6 K 2023-04-25 02:56:06,243 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/.tmp/C/4b534914c5064cea8af561d5965651c6 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/C/4b534914c5064cea8af561d5965651c6 2023-04-25 02:56:06,246 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/C/4b534914c5064cea8af561d5965651c6, entries=150, sequenceid=68, filesize=11.6 K 2023-04-25 02:56:06,247 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for d6e38fead90b2a328d380bf5577ff0ce in 466ms, sequenceid=68, compaction requested=true 2023-04-25 02:56:06,247 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d6e38fead90b2a328d380bf5577ff0ce: 2023-04-25 02:56:06,247 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. 2023-04-25 02:56:06,247 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:56:06,247 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:56:06,248 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:06,248 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:56:06,248 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:56:06,249 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:06,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:06,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:06,249 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:06,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:06,250 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:06,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:06,250 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:56:06,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:06,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:06,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:06,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:06,252 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:56:06,252 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5376ff75[Count = 0] remaining members to acquire global barrier 2023-04-25 02:56:06,252 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:56:06,252 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:06,254 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:06,254 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:06,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:06,254 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:56:06,254 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:56:06,254 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:56:06,254 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:06,254 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:56:06,256 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:06,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:06,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:06,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:06,256 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:56:06,256 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:56:06,256 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:56:06,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:06,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:06,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:06,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:06,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:06,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:06,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:06,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:56:06,258 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:56:06,258 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:56:06,258 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:56:06,259 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:56:06,259 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:56:06,259 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:56:06,260 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:06,260 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:06,260 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:06,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:06,260 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:56:06,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:06,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:06,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:06,261 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:56:06,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:56:06,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:06,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:06,261 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:06,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:06,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:06,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:06,263 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:06,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:06,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:06,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:06,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:06,269 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:56:06,269 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:56:06,269 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:56:06,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:56:06,269 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:06,269 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:06,269 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:06,269 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:06,269 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:06,269 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:56:06,269 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:56:06,270 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:56:06,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:56:06,269 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:06,270 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:56:06,270 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:56:06,270 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:06,270 DEBUG [Thread-3656] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:56:06,270 DEBUG [Thread-3656] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:56:06,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(9158): Flush requested on d6e38fead90b2a328d380bf5577ff0ce 2023-04-25 02:56:06,319 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing d6e38fead90b2a328d380bf5577ff0ce 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-25 02:56:06,320 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6e38fead90b2a328d380bf5577ff0ce, store=A 2023-04-25 02:56:06,320 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:56:06,320 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6e38fead90b2a328d380bf5577ff0ce, store=B 2023-04-25 02:56:06,320 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:56:06,320 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6e38fead90b2a328d380bf5577ff0ce, store=C 2023-04-25 02:56:06,320 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:56:06,350 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682391366335, max timestamp=1682391366348Num uniques 859; 2023-04-25 02:56:06,350 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-25 02:56:06,351 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682391366335, max timestamp=1682391366348Num uniques 859; 2023-04-25 02:56:06,351 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-25 02:56:06,352 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682391366335, max timestamp=1682391366348Num uniques 859; 2023-04-25 02:56:06,352 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-25 02:56:06,370 DEBUG [Thread-3656] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:56:06,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:56:06,373 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:56:06,375 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:06,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39526 deadline: 1682391426369, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:06,376 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:06,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39522 deadline: 1682391426370, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:06,380 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:06,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39482 deadline: 1682391426375, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:06,380 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:06,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39536 deadline: 1682391426376, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:06,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:56:06,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:56:06,384 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:56:06,384 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:56:06,384 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:56:06,384 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:56:06,385 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:06,385 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:06,386 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:06,386 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:56:06,386 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:56:06,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:56:06,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:06,387 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:06,387 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:06,387 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:56:06,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:56:06,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:06,387 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:56:06,388 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:56:06,388 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:56:06,388 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:56:06,388 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:56:06,388 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:56:06,391 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:56:06,391 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:56:06,391 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. 2023-04-25 02:56:06,391 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. started... 2023-04-25 02:56:06,391 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. as already flushing 2023-04-25 02:56:06,391 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. 2023-04-25 02:56:06,391 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:56:06,391 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:56:06,391 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:06,391 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:56:06,391 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:56:06,393 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:06,393 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:06,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:06,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:06,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:06,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:06,393 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:06,393 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:56:06,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:06,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:06,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:06,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:06,395 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:56:06,395 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6127f037[Count = 0] remaining members to acquire global barrier 2023-04-25 02:56:06,396 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:56:06,396 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:06,397 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:06,397 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:06,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:06,397 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:56:06,397 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:56:06,397 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:56:06,398 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:06,398 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:56:06,399 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:06,399 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:56:06,399 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:56:06,399 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:56:06,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:06,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:06,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:06,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:06,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:06,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:06,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:06,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:06,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:06,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:06,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:56:06,402 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:56:06,402 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:56:06,402 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:56:06,402 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:56:06,402 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:56:06,402 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:56:06,405 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:06,405 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:56:06,406 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:06,406 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:06,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:06,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:06,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:06,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:06,406 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:56:06,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:56:06,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:06,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:06,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:06,407 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:06,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:06,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:06,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:06,409 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:06,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:06,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:06,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:06,414 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:56:06,414 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:56:06,414 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:56:06,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:56:06,414 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:06,414 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:06,414 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:56:06,414 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:56:06,414 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:56:06,414 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:06,415 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:06,415 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:06,415 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:06,415 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:06,415 DEBUG [Thread-3656] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:56:06,415 DEBUG [Thread-3656] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:56:06,415 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:56:06,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:56:06,415 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:56:06,480 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:06,480 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39526 deadline: 1682391426477, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:06,480 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:06,480 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39522 deadline: 1682391426477, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:06,484 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:06,484 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:06,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39482 deadline: 1682391426481, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:06,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39536 deadline: 1682391426481, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:06,515 DEBUG [Thread-3656] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:56:06,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:56:06,517 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:56:06,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:56:06,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:56:06,518 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:56:06,519 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:56:06,519 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:56:06,519 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:56:06,520 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:06,520 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:06,521 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:06,521 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:56:06,521 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:56:06,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:56:06,521 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:06,521 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:56:06,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:06,522 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:06,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:56:06,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:06,522 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:56:06,522 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:56:06,522 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:56:06,522 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:56:06,522 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:56:06,522 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:56:06,523 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:56:06,523 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:56:06,523 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. 2023-04-25 02:56:06,523 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. started... 2023-04-25 02:56:06,523 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. as already flushing 2023-04-25 02:56:06,523 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. 2023-04-25 02:56:06,523 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:56:06,523 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:56:06,523 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:06,523 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:56:06,523 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:56:06,526 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:06,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:06,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:06,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:06,526 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:06,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:06,527 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:06,527 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:56:06,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:06,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:06,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:06,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:06,528 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:56:06,528 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6116ea19[Count = 0] remaining members to acquire global barrier 2023-04-25 02:56:06,528 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:56:06,528 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:06,529 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:06,529 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:06,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:06,529 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:56:06,529 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:06,529 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:56:06,529 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:56:06,529 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:56:06,531 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:06,531 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:56:06,531 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:56:06,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:06,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:06,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:06,531 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:56:06,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:06,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:06,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:06,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:06,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:06,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:06,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:06,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:56:06,533 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:56:06,533 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:56:06,533 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:56:06,533 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:56:06,533 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:56:06,533 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:56:06,535 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:06,535 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:06,535 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:06,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:06,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:06,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:06,535 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:56:06,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:06,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:06,536 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:56:06,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:56:06,536 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:06,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:06,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:06,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:06,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:06,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:06,538 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:06,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:06,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:06,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:06,541 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:06,541 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:06,541 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:06,541 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:06,541 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:56:06,541 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:56:06,541 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:56:06,541 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:56:06,541 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:06,541 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:56:06,541 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:06,541 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:56:06,541 DEBUG [Thread-3656] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:56:06,541 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:56:06,541 DEBUG [Thread-3656] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:56:06,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:56:06,541 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:06,542 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:56:06,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:56:06,561 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:06,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39458 deadline: 1682391426559, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:06,642 DEBUG [Thread-3656] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:56:06,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:56:06,643 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:56:06,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:56:06,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:56:06,645 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:56:06,645 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:56:06,645 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:56:06,645 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:56:06,645 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:06,645 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:06,646 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:56:06,646 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:06,646 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:56:06,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:56:06,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:06,647 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:06,647 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:56:06,647 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:06,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:56:06,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:06,647 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:56:06,647 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:56:06,648 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:56:06,648 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:56:06,648 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:56:06,648 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:56:06,648 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:56:06,648 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:56:06,648 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. 2023-04-25 02:56:06,648 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. started... 2023-04-25 02:56:06,648 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. as already flushing 2023-04-25 02:56:06,648 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. 2023-04-25 02:56:06,648 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:56:06,648 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:56:06,648 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:06,648 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:56:06,648 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:56:06,649 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:06,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:06,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:06,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:06,649 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:06,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:06,650 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:06,650 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:56:06,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:06,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:06,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:06,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:06,651 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:56:06,651 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@a71e5b1[Count = 0] remaining members to acquire global barrier 2023-04-25 02:56:06,651 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:56:06,651 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:06,652 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:06,652 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:06,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:06,652 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:56:06,652 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:56:06,652 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:56:06,652 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:06,652 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:56:06,654 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:06,654 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:56:06,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:06,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:06,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:06,654 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:56:06,654 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:56:06,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:06,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:06,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:06,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:06,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:06,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:06,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:06,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:56:06,657 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:56:06,657 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:56:06,657 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:56:06,657 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:56:06,657 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:56:06,657 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:56:06,658 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:06,658 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:06,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:06,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:06,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:06,658 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:56:06,658 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:06,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:06,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:06,658 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:06,659 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:56:06,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:56:06,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:06,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:06,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:06,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:06,660 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:06,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:06,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:06,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:06,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:06,668 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:06,668 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:06,668 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:06,668 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:56:06,668 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:56:06,668 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:56:06,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:56:06,668 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:06,668 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:06,668 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:56:06,668 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:56:06,668 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:56:06,668 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:06,668 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:06,668 DEBUG [Thread-3656] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:56:06,668 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:56:06,668 DEBUG [Thread-3656] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:56:06,669 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:56:06,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:56:06,684 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:06,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39526 deadline: 1682391426682, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:06,684 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:06,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39522 deadline: 1682391426682, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:06,689 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:06,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39482 deadline: 1682391426685, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:06,690 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:06,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39536 deadline: 1682391426686, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:06,769 DEBUG [Thread-3656] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:56:06,769 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:56:06,771 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230425e31609ee9b7649089fdbc27e6fbf483a to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425e31609ee9b7649089fdbc27e6fbf483a 2023-04-25 02:56:06,771 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:56:06,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:56:06,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:56:06,773 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:56:06,773 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:56:06,773 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:56:06,773 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:56:06,774 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:06,774 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:06,776 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:56:06,776 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:06,776 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:56:06,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:56:06,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:06,777 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:06,777 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:56:06,777 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:06,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:56:06,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:06,778 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:56:06,778 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:56:06,778 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:56:06,778 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:56:06,778 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:56:06,778 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:56:06,778 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:56:06,778 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:56:06,778 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. 2023-04-25 02:56:06,778 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. started... 2023-04-25 02:56:06,778 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. as already flushing 2023-04-25 02:56:06,779 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. 2023-04-25 02:56:06,779 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:56:06,779 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:56:06,779 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:06,779 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:56:06,779 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:56:06,780 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:06,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:06,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:06,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:06,780 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:06,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:06,781 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:06,781 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:56:06,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:06,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:06,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:06,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:06,782 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:56:06,782 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6e982edd[Count = 0] remaining members to acquire global barrier 2023-04-25 02:56:06,782 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:56:06,782 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:06,783 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:06,783 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:06,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:06,783 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:56:06,783 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:56:06,783 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:56:06,783 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:06,784 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:56:06,785 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:06,785 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:56:06,785 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:56:06,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:06,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:06,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:06,785 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:56:06,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:06,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:06,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:06,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:06,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:06,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:06,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:06,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:56:06,788 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:56:06,788 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:56:06,788 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:56:06,788 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:56:06,788 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:56:06,788 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:56:06,790 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:06,790 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:56:06,790 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:06,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:06,790 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:06,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:06,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:06,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:06,791 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:56:06,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:56:06,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:06,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:06,791 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:06,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:06,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:06,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:06,792 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:06,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:06,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:06,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:06,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:06,795 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:56:06,795 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:56:06,795 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:56:06,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:56:06,795 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:06,795 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:06,795 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:06,795 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:56:06,795 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:56:06,795 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:56:06,795 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:56:06,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:56:06,795 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:06,796 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:06,796 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:06,796 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:06,796 DEBUG [Thread-3656] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:56:06,796 DEBUG [Thread-3656] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:56:06,896 DEBUG [Thread-3656] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:56:06,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:56:06,898 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:56:06,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:56:06,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:56:06,900 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:56:06,900 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:56:06,900 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:56:06,900 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:56:06,900 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:06,900 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:06,901 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:56:06,901 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:06,901 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:56:06,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:56:06,902 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:06,902 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:56:06,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:06,902 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:06,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:56:06,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:06,902 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:56:06,903 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:56:06,903 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:56:06,903 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:56:06,903 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:56:06,903 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:56:06,903 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:56:06,903 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:56:06,903 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. 2023-04-25 02:56:06,903 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. started... 2023-04-25 02:56:06,903 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. as already flushing 2023-04-25 02:56:06,903 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. 2023-04-25 02:56:06,903 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:56:06,903 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:56:06,903 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:06,903 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:56:06,903 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:56:06,905 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:06,905 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:06,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:06,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:06,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:06,905 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:06,905 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:56:06,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:06,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:06,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:06,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:06,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:06,906 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:56:06,906 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@30e0dd71[Count = 0] remaining members to acquire global barrier 2023-04-25 02:56:06,906 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:56:06,907 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:06,909 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:06,909 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:06,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:06,909 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:56:06,909 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:56:06,909 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:06,909 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:56:06,909 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:56:06,910 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:06,910 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:56:06,910 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:56:06,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:06,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:06,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:06,910 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:56:06,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:06,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:06,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:06,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:06,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:06,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:06,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:06,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:56:06,913 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:56:06,913 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:56:06,913 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:56:06,913 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:56:06,913 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:56:06,913 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:56:06,915 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:06,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:06,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:06,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:06,915 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:06,915 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:06,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:06,915 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:06,915 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:56:06,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:06,916 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:56:06,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:56:06,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:06,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:06,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:06,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:06,917 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:06,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:06,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:06,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:06,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:06,919 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:56:06,919 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:06,919 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:56:06,919 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:56:06,919 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:56:06,919 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:06,919 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:06,919 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:06,919 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:06,919 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:06,919 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:06,919 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:56:06,919 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:56:06,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:56:06,919 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:56:06,919 DEBUG [Thread-3656] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:56:06,920 DEBUG [Thread-3656] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:56:06,920 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:56:06,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:56:06,989 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:06,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39526 deadline: 1682391426987, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:06,989 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:06,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39522 deadline: 1682391426987, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:06,992 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:06,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39482 deadline: 1682391426991, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:06,995 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:06,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39536 deadline: 1682391426993, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:07,020 DEBUG [Thread-3656] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:56:07,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:56:07,022 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:56:07,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:56:07,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:56:07,023 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:56:07,023 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:56:07,023 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:56:07,023 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:56:07,024 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:07,024 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:07,027 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:07,027 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:56:07,027 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:56:07,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:56:07,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:07,027 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:07,027 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:56:07,028 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:07,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:56:07,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:07,028 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:56:07,028 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:56:07,028 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:56:07,028 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:56:07,028 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:56:07,028 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:56:07,029 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:56:07,029 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:56:07,029 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. 2023-04-25 02:56:07,029 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. started... 2023-04-25 02:56:07,029 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. as already flushing 2023-04-25 02:56:07,029 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. 2023-04-25 02:56:07,029 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:56:07,029 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:56:07,029 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:07,029 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:56:07,029 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:56:07,031 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:07,031 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:07,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:07,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:07,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:07,031 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:07,031 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:56:07,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:07,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:07,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:07,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:07,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:07,032 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:56:07,032 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6670bf56[Count = 0] remaining members to acquire global barrier 2023-04-25 02:56:07,032 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:56:07,033 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:07,034 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:07,034 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:07,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:07,034 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:56:07,034 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:56:07,034 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:56:07,034 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:07,034 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:56:07,035 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:56:07,035 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:07,035 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:56:07,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:07,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:07,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:07,036 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:56:07,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:07,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:07,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:07,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:07,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:07,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:07,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:07,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:56:07,038 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:56:07,038 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:56:07,038 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:56:07,038 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:56:07,039 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:56:07,039 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:56:07,042 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:07,042 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:56:07,042 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:07,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:07,042 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:07,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:07,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:07,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:07,043 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:56:07,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:56:07,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:07,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:07,043 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:07,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:07,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:07,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:07,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:07,044 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:07,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:07,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:07,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:07,046 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:56:07,046 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:56:07,046 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:56:07,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:56:07,046 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:07,046 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:56:07,047 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:56:07,046 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:07,047 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:07,047 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:07,046 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:56:07,047 DEBUG [Thread-3656] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:56:07,047 DEBUG [Thread-3656] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:56:07,047 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:07,047 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:56:07,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:56:07,047 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:07,047 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:07,147 DEBUG [Thread-3656] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:56:07,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:56:07,149 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:56:07,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:56:07,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:56:07,151 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:56:07,151 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:56:07,151 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:56:07,151 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:56:07,151 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:07,152 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:07,153 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:07,153 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:56:07,153 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:56:07,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:56:07,153 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:07,153 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:56:07,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:07,154 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:07,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:56:07,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:07,154 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:56:07,154 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:56:07,154 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:56:07,154 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:56:07,154 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:56:07,154 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:56:07,154 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:56:07,155 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:56:07,155 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. 2023-04-25 02:56:07,155 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. started... 2023-04-25 02:56:07,155 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. as already flushing 2023-04-25 02:56:07,155 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. 2023-04-25 02:56:07,155 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:56:07,155 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:56:07,155 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:07,155 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:56:07,155 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:56:07,160 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:07,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:07,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:07,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:07,161 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:07,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:07,161 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:07,161 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:56:07,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:07,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:07,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:07,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:07,163 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:56:07,163 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@63299244[Count = 0] remaining members to acquire global barrier 2023-04-25 02:56:07,163 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:56:07,163 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:07,164 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:07,164 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:07,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:07,164 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:07,164 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:56:07,164 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:56:07,165 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:56:07,165 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:56:07,166 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:56:07,166 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:56:07,166 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:07,166 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:56:07,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:07,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:07,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:07,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:07,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:07,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:07,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:07,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:07,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:07,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:07,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:56:07,169 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:56:07,169 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:56:07,169 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:56:07,169 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:56:07,169 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:56:07,169 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:56:07,170 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:07,170 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:07,170 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:07,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:07,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:07,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:07,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:07,170 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:56:07,171 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:56:07,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:56:07,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:07,171 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:07,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:07,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:07,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:07,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:07,172 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:07,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:07,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:07,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:07,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:07,175 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:07,176 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:07,176 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:07,176 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:56:07,176 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:56:07,176 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:56:07,176 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:56:07,176 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:56:07,176 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:56:07,176 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:56:07,176 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:07,176 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:07,176 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:07,176 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:07,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:56:07,176 DEBUG [Thread-3656] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:56:07,176 DEBUG [Thread-3656] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:56:07,177 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:56:07,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:56:07,177 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=81, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/.tmp/A/117b1fbefdb546d88cdb47a4df92e8de 2023-04-25 02:56:07,276 DEBUG [Thread-3656] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:56:07,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:56:07,278 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:56:07,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:56:07,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:56:07,280 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:56:07,280 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:56:07,280 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:56:07,280 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:56:07,280 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:07,280 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:07,282 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:56:07,282 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:56:07,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:56:07,282 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:07,282 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:07,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:07,282 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:56:07,282 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:07,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:56:07,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:07,283 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:56:07,283 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:56:07,283 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:56:07,283 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:56:07,283 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:56:07,283 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:56:07,283 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:56:07,284 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:56:07,284 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. 2023-04-25 02:56:07,284 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. started... 2023-04-25 02:56:07,284 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. as already flushing 2023-04-25 02:56:07,284 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. 2023-04-25 02:56:07,284 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:56:07,284 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:56:07,284 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:07,284 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:56:07,284 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:56:07,285 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:07,285 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:07,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:07,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:07,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:07,286 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:07,286 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:56:07,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:07,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:07,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:07,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:07,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:07,287 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:56:07,287 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4ce8a63a[Count = 0] remaining members to acquire global barrier 2023-04-25 02:56:07,287 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:56:07,287 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:07,288 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:07,288 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:07,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:07,289 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:56:07,289 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:56:07,289 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:07,289 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:56:07,289 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:56:07,291 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:07,291 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:56:07,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:07,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:07,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:07,291 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:56:07,291 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:56:07,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:07,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:07,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:07,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:07,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:07,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:07,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:07,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:56:07,294 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:56:07,294 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:56:07,294 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:56:07,294 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:56:07,294 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:56:07,294 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:56:07,295 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:07,295 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:07,295 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:07,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:07,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:07,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:07,295 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:56:07,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:07,296 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:56:07,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:56:07,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:07,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:07,299 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:07,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:07,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:07,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:07,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:07,300 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:07,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:07,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:07,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:07,303 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:56:07,303 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:56:07,303 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:56:07,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:56:07,303 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:07,303 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:07,303 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:07,303 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:07,303 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:56:07,304 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:56:07,304 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:56:07,303 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:07,304 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:07,304 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:07,304 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:56:07,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:56:07,304 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:56:07,305 DEBUG [Thread-3656] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:56:07,305 DEBUG [Thread-3656] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:56:07,405 DEBUG [Thread-3656] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:56:07,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:56:07,408 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:56:07,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:56:07,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:56:07,409 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:56:07,409 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:56:07,409 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:56:07,409 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:56:07,409 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:07,409 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:07,410 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:56:07,411 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:07,411 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:56:07,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:56:07,411 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:07,411 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:56:07,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:07,411 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:07,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:56:07,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:07,412 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:56:07,412 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:56:07,412 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:56:07,412 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:56:07,412 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:56:07,412 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:56:07,412 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:56:07,412 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:56:07,412 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. 2023-04-25 02:56:07,412 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. started... 2023-04-25 02:56:07,413 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. as already flushing 2023-04-25 02:56:07,413 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. 2023-04-25 02:56:07,413 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:56:07,413 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:56:07,413 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:07,413 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:56:07,413 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:56:07,415 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:07,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:07,415 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:07,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:07,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:07,416 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:07,416 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:56:07,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:07,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:07,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:07,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:07,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:07,417 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:56:07,417 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7ad5e183[Count = 0] remaining members to acquire global barrier 2023-04-25 02:56:07,417 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:56:07,417 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:07,418 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:07,418 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:07,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:07,418 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:56:07,418 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:56:07,418 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:56:07,418 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:07,418 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:56:07,419 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:07,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:07,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:07,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:07,420 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:56:07,420 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:56:07,420 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:56:07,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:07,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:07,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:07,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:07,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:07,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:07,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:07,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:56:07,422 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:56:07,422 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:56:07,422 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:56:07,422 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:56:07,422 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:56:07,422 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:56:07,423 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:07,423 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:07,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:07,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:07,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:07,423 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:07,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:07,423 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:56:07,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:07,424 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:56:07,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:56:07,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:07,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:07,425 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:07,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:07,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:07,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:07,425 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:07,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:07,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:07,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:07,428 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:07,428 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:56:07,428 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:07,428 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:56:07,428 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:56:07,428 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:56:07,428 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:56:07,429 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:56:07,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:56:07,428 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:07,429 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:07,429 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:07,429 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:07,429 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:07,429 DEBUG [Thread-3656] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:56:07,429 DEBUG [Thread-3656] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:56:07,429 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:56:07,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:56:07,494 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:07,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39526 deadline: 1682391427490, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:07,495 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:07,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39522 deadline: 1682391427493, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:07,499 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:07,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39482 deadline: 1682391427495, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:07,500 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:07,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39536 deadline: 1682391427498, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:07,529 DEBUG [Thread-3656] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:56:07,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:56:07,531 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:56:07,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:56:07,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:56:07,533 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:56:07,533 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:56:07,533 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:56:07,533 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:56:07,533 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:07,533 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:07,534 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:56:07,534 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:07,534 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:56:07,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:56:07,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:07,535 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:07,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:56:07,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:07,536 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:56:07,536 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:56:07,536 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:56:07,536 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:07,536 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:56:07,536 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:56:07,536 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:56:07,536 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:56:07,537 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:56:07,537 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:56:07,537 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. 2023-04-25 02:56:07,537 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. started... 2023-04-25 02:56:07,537 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. as already flushing 2023-04-25 02:56:07,537 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. 2023-04-25 02:56:07,537 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:56:07,537 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:56:07,537 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:07,537 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:56:07,537 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:56:07,539 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:07,539 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:07,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:07,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:07,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:07,539 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:07,539 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:56:07,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:07,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:07,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:07,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:07,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:07,541 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:56:07,541 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@193c7b80[Count = 0] remaining members to acquire global barrier 2023-04-25 02:56:07,541 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:56:07,541 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:07,545 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:07,545 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:07,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:07,545 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:56:07,546 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:56:07,546 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:56:07,546 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:07,546 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:56:07,547 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:07,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:07,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:07,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:07,547 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:56:07,547 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:56:07,547 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:56:07,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:07,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:07,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:07,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:07,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:07,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:07,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:07,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:56:07,550 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:56:07,550 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:56:07,550 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:56:07,550 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:56:07,550 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:56:07,550 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:56:07,551 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:07,551 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:07,551 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:56:07,551 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:07,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:07,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:07,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:07,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:07,552 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:56:07,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:56:07,552 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:07,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:07,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:07,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:07,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:07,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:07,554 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:07,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:07,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:07,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:07,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:07,556 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:07,556 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:56:07,556 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:07,556 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:56:07,556 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:07,556 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:56:07,557 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:56:07,556 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:56:07,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:56:07,556 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:56:07,556 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:07,557 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:07,557 DEBUG [Thread-3656] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:56:07,557 DEBUG [Thread-3656] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:56:07,557 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:07,557 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:07,557 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:56:07,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:56:07,588 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=81 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/.tmp/B/86fa3099f8df418eb977b338ee9a1a0a 2023-04-25 02:56:07,598 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=81 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/.tmp/C/0d834cf5a8b74020a1839f1b56143199 2023-04-25 02:56:07,602 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/.tmp/A/117b1fbefdb546d88cdb47a4df92e8de as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/A/117b1fbefdb546d88cdb47a4df92e8de 2023-04-25 02:56:07,605 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/A/117b1fbefdb546d88cdb47a4df92e8de, entries=150, sequenceid=81, filesize=25.1 K 2023-04-25 02:56:07,606 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/.tmp/B/86fa3099f8df418eb977b338ee9a1a0a as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/B/86fa3099f8df418eb977b338ee9a1a0a 2023-04-25 02:56:07,609 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/B/86fa3099f8df418eb977b338ee9a1a0a, entries=100, sequenceid=81, filesize=9.3 K 2023-04-25 02:56:07,610 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/.tmp/C/0d834cf5a8b74020a1839f1b56143199 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/C/0d834cf5a8b74020a1839f1b56143199 2023-04-25 02:56:07,613 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/C/0d834cf5a8b74020a1839f1b56143199, entries=100, sequenceid=81, filesize=9.3 K 2023-04-25 02:56:07,614 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=114.05 KB/116790 for d6e38fead90b2a328d380bf5577ff0ce in 1295ms, sequenceid=81, compaction requested=true 2023-04-25 02:56:07,614 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for d6e38fead90b2a328d380bf5577ff0ce: 2023-04-25 02:56:07,614 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 02:56:07,614 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-25 02:56:07,615 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 02:56:07,615 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-25 02:56:07,615 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 02:56:07,616 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 128726 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-25 02:56:07,616 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HStore(1912): d6e38fead90b2a328d380bf5577ff0ce/A is initiating minor compaction (all files) 2023-04-25 02:56:07,616 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 57086 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-25 02:56:07,616 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of d6e38fead90b2a328d380bf5577ff0ce/A in TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. 2023-04-25 02:56:07,616 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HStore(1912): d6e38fead90b2a328d380bf5577ff0ce/B is initiating minor compaction (all files) 2023-04-25 02:56:07,616 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/A/dfb20049ffd144eca1334779fe1d1e28, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/A/969bc1313d61424db21ef2e2fcf5c054, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/A/c269266e65534535b43d63130a94f865, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/A/6e508d77291c4970bbe3d5ea0c612ece, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/A/117b1fbefdb546d88cdb47a4df92e8de] into tmpdir=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/.tmp, totalSize=125.7 K 2023-04-25 02:56:07,616 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d6e38fead90b2a328d380bf5577ff0ce/B in TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. 2023-04-25 02:56:07,616 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/B/45c27d721ee4419aaf06aae8aa2eec83, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/B/0d44dc3aecf34e1595d7ef4216834a8f, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/B/777035c54c8f48b4b5899b5d576af640, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/B/13aa7915d1df47d6909e42a1c60402a3, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/B/86fa3099f8df418eb977b338ee9a1a0a] into tmpdir=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/.tmp, totalSize=55.7 K 2023-04-25 02:56:07,616 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting dfb20049ffd144eca1334779fe1d1e28, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=16, earliestPutTs=1682391363240 2023-04-25 02:56:07,617 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting 969bc1313d61424db21ef2e2fcf5c054, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=36, earliestPutTs=1682391363262 2023-04-25 02:56:07,617 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting 45c27d721ee4419aaf06aae8aa2eec83, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=16, earliestPutTs=1682391363240 2023-04-25 02:56:07,617 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting c269266e65534535b43d63130a94f865, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=48, earliestPutTs=1682391363879 2023-04-25 02:56:07,617 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting 0d44dc3aecf34e1595d7ef4216834a8f, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=36, earliestPutTs=1682391363262 2023-04-25 02:56:07,618 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting 6e508d77291c4970bbe3d5ea0c612ece, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=68, earliestPutTs=1682391364428 2023-04-25 02:56:07,618 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting 777035c54c8f48b4b5899b5d576af640, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=48, earliestPutTs=1682391363879 2023-04-25 02:56:07,618 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting 117b1fbefdb546d88cdb47a4df92e8de, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=81, earliestPutTs=1682391366006 2023-04-25 02:56:07,618 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting 13aa7915d1df47d6909e42a1c60402a3, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=68, earliestPutTs=1682391364428 2023-04-25 02:56:07,618 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting 86fa3099f8df418eb977b338ee9a1a0a, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=81, earliestPutTs=1682391366009 2023-04-25 02:56:07,626 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] throttle.PressureAwareThroughputController(145): d6e38fead90b2a328d380bf5577ff0ce#B#compaction#499 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 53.85 MB/second 2023-04-25 02:56:07,633 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] throttle.PressureAwareThroughputController(145): d6e38fead90b2a328d380bf5577ff0ce#A#compaction#498 average throughput is 2.19 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-25 02:56:07,643 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/.tmp/B/a0d9a6ee1308466f8489abd8adc7ded5 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/B/a0d9a6ee1308466f8489abd8adc7ded5 2023-04-25 02:56:07,648 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in d6e38fead90b2a328d380bf5577ff0ce/B of d6e38fead90b2a328d380bf5577ff0ce into a0d9a6ee1308466f8489abd8adc7ded5(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 02:56:07,648 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d6e38fead90b2a328d380bf5577ff0ce: 2023-04-25 02:56:07,648 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce., storeName=d6e38fead90b2a328d380bf5577ff0ce/B, priority=11, startTime=1682391367614; duration=0sec 2023-04-25 02:56:07,648 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 02:56:07,648 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-25 02:56:07,650 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 57086 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-25 02:56:07,650 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HStore(1912): d6e38fead90b2a328d380bf5577ff0ce/C is initiating minor compaction (all files) 2023-04-25 02:56:07,650 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d6e38fead90b2a328d380bf5577ff0ce/C in TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. 2023-04-25 02:56:07,650 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/C/24dc7dbdd2dc4078bb27b4eb8de7f60f, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/C/5bec0d8e724d4b79a35ecf5d0a9c9b58, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/C/3d3f3b9b61a84f8db4adefcfef0d6cad, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/C/4b534914c5064cea8af561d5965651c6, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/C/0d834cf5a8b74020a1839f1b56143199] into tmpdir=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/.tmp, totalSize=55.7 K 2023-04-25 02:56:07,650 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting 24dc7dbdd2dc4078bb27b4eb8de7f60f, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=16, earliestPutTs=1682391363240 2023-04-25 02:56:07,651 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting 5bec0d8e724d4b79a35ecf5d0a9c9b58, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=36, earliestPutTs=1682391363262 2023-04-25 02:56:07,651 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting 3d3f3b9b61a84f8db4adefcfef0d6cad, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=48, earliestPutTs=1682391363879 2023-04-25 02:56:07,651 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting 4b534914c5064cea8af561d5965651c6, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=68, earliestPutTs=1682391364428 2023-04-25 02:56:07,652 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting 0d834cf5a8b74020a1839f1b56143199, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=81, earliestPutTs=1682391366009 2023-04-25 02:56:07,657 DEBUG [Thread-3656] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:56:07,657 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] throttle.PressureAwareThroughputController(145): d6e38fead90b2a328d380bf5577ff0ce#C#compaction#500 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-25 02:56:07,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:56:07,660 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:56:07,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:56:07,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:56:07,664 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:56:07,664 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:56:07,664 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:56:07,664 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:56:07,664 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:07,664 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:07,666 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:07,666 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:56:07,666 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:56:07,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:56:07,667 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:07,667 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:56:07,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:07,667 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:07,667 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/.tmp/C/bc39437696f2440cba5221a588d9b824 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/C/bc39437696f2440cba5221a588d9b824 2023-04-25 02:56:07,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:56:07,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:07,668 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:56:07,668 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:56:07,668 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:56:07,668 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:56:07,668 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:56:07,669 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:56:07,671 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in d6e38fead90b2a328d380bf5577ff0ce/C of d6e38fead90b2a328d380bf5577ff0ce into bc39437696f2440cba5221a588d9b824(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 02:56:07,671 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d6e38fead90b2a328d380bf5577ff0ce: 2023-04-25 02:56:07,671 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce., storeName=d6e38fead90b2a328d380bf5577ff0ce/C, priority=11, startTime=1682391367615; duration=0sec 2023-04-25 02:56:07,671 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 02:56:07,672 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:56:07,672 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:56:07,672 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. 2023-04-25 02:56:07,672 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. started... 2023-04-25 02:56:07,672 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d6e38fead90b2a328d380bf5577ff0ce 3/3 column families, dataSize=114.05 KB heapSize=315.84 KB 2023-04-25 02:56:07,672 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6e38fead90b2a328d380bf5577ff0ce, store=A 2023-04-25 02:56:07,672 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 02:56:07,672 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6e38fead90b2a328d380bf5577ff0ce, store=B 2023-04-25 02:56:07,672 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 02:56:07,672 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6e38fead90b2a328d380bf5577ff0ce, store=C 2023-04-25 02:56:07,672 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 02:56:07,701 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230425514f169034b7481b8ac68acd2aa40dd8 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425514f169034b7481b8ac68acd2aa40dd8 2023-04-25 02:56:07,717 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=115, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/.tmp/A/6d439419060d4cd4a5bb0300afe993c6 2023-04-25 02:56:07,728 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=115 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/.tmp/B/2ea6f369063149b38f4755ea9d2aaa4a 2023-04-25 02:56:07,741 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=115 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/.tmp/C/ac631873d4384b54a0d9595795e44b24 2023-04-25 02:56:07,746 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/.tmp/A/6d439419060d4cd4a5bb0300afe993c6 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/A/6d439419060d4cd4a5bb0300afe993c6 2023-04-25 02:56:07,749 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/A/6d439419060d4cd4a5bb0300afe993c6, entries=150, sequenceid=115, filesize=25.1 K 2023-04-25 02:56:07,750 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/.tmp/B/2ea6f369063149b38f4755ea9d2aaa4a as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/B/2ea6f369063149b38f4755ea9d2aaa4a 2023-04-25 02:56:07,753 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/B/2ea6f369063149b38f4755ea9d2aaa4a, entries=150, sequenceid=115, filesize=11.6 K 2023-04-25 02:56:07,754 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/.tmp/C/ac631873d4384b54a0d9595795e44b24 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/C/ac631873d4384b54a0d9595795e44b24 2023-04-25 02:56:07,757 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/C/ac631873d4384b54a0d9595795e44b24, entries=150, sequenceid=115, filesize=11.6 K 2023-04-25 02:56:07,757 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~315.80 KB/323376, currentSize=0 B/0 for d6e38fead90b2a328d380bf5577ff0ce in 85ms, sequenceid=115, compaction requested=false 2023-04-25 02:56:07,757 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d6e38fead90b2a328d380bf5577ff0ce: 2023-04-25 02:56:07,758 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. 2023-04-25 02:56:07,758 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:56:07,758 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:56:07,758 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:07,758 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:56:07,758 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:56:07,759 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:07,759 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:07,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:07,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:07,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:07,760 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:07,760 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:56:07,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:07,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:07,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:07,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:07,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:07,761 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:56:07,762 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@36286151[Count = 0] remaining members to acquire global barrier 2023-04-25 02:56:07,762 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:56:07,762 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:07,763 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:07,763 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:07,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:07,763 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:56:07,763 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:56:07,763 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:56:07,763 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:07,763 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:56:07,764 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:56:07,764 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:56:07,764 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:56:07,768 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:07,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:07,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:07,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:07,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:07,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:07,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:07,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:07,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:07,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:07,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:07,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:56:07,778 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:56:07,778 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:56:07,778 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:56:07,778 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:56:07,779 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:56:07,779 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:56:07,781 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:07,781 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:07,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:07,781 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:07,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:07,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:07,781 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:56:07,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:07,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:07,781 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:07,781 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:56:07,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:56:07,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:07,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:07,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:07,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:07,783 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:07,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:07,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:07,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:07,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:07,797 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:07,797 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:56:07,798 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:56:07,798 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:56:07,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:56:07,797 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:07,798 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:56:07,798 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:56:07,798 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:56:07,798 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:56:07,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:56:07,798 DEBUG [Thread-3656] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:56:07,798 DEBUG [Thread-3656] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:56:07,798 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:07,798 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:56:07,798 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:07,798 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:07,798 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:07,798 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:07,898 DEBUG [Thread-3656] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:56:07,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:56:07,900 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:56:07,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:56:07,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:56:07,901 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:56:07,901 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:56:07,901 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:56:07,901 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:56:07,902 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:07,902 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:07,903 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:56:07,903 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:07,903 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:56:07,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:56:07,903 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:07,903 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:56:07,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:07,904 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:07,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:56:07,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:07,904 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:56:07,904 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:56:07,904 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:56:07,905 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:56:07,905 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:56:07,905 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:56:07,905 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:56:07,905 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. 2023-04-25 02:56:07,905 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. started... 2023-04-25 02:56:07,905 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d6e38fead90b2a328d380bf5577ff0ce: 2023-04-25 02:56:07,905 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. 2023-04-25 02:56:07,905 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:56:07,905 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:56:07,905 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:07,905 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:56:07,905 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:56:07,906 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:07,906 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:07,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:07,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:07,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:07,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:07,907 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:07,907 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:56:07,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:07,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:07,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:07,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:07,908 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:56:07,908 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3f78c8d8[Count = 0] remaining members to acquire global barrier 2023-04-25 02:56:07,908 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:56:07,908 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:07,910 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:07,910 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:07,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:07,911 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:56:07,911 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:56:07,911 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:07,911 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:56:07,911 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:56:07,912 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:07,912 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:56:07,912 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:56:07,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:07,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:07,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:07,912 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:56:07,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:07,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:07,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:07,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:07,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:07,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:07,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:07,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:56:07,914 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:56:07,914 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:56:07,915 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:56:07,915 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:56:07,915 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:56:07,915 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:56:07,916 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:07,916 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:07,916 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:07,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:07,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:07,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:07,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:07,916 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:56:07,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:07,917 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:56:07,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:56:07,917 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:07,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:07,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:07,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:07,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:07,918 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:07,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:07,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:07,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:07,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:07,921 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:07,921 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:07,921 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:07,921 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:07,921 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:07,921 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:07,921 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:07,921 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:56:07,921 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:56:07,921 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:56:07,921 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:56:07,921 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:56:07,921 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:56:07,922 DEBUG [Thread-3656] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:56:07,922 DEBUG [Thread-3656] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:56:07,921 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:56:07,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:56:07,922 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:56:07,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:56:08,022 DEBUG [Thread-3656] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:56:08,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:56:08,024 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:56:08,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:56:08,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:56:08,025 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:56:08,025 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:56:08,025 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:56:08,026 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:56:08,026 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:08,026 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:08,028 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:56:08,028 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:08,028 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:56:08,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:56:08,028 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:08,028 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:56:08,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:08,029 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:08,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:56:08,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:08,029 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:56:08,029 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:56:08,029 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:56:08,030 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:56:08,030 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:56:08,030 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:56:08,030 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:56:08,030 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:56:08,030 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. 2023-04-25 02:56:08,030 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. started... 2023-04-25 02:56:08,030 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d6e38fead90b2a328d380bf5577ff0ce: 2023-04-25 02:56:08,030 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. 2023-04-25 02:56:08,030 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:56:08,030 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:56:08,030 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:08,031 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:56:08,031 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:56:08,032 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:08,032 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:08,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:08,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:08,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:08,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:08,032 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:08,033 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:56:08,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:08,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:08,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:08,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:08,034 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:56:08,034 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6af4ff4d[Count = 0] remaining members to acquire global barrier 2023-04-25 02:56:08,034 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:56:08,034 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:08,035 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:08,035 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:08,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:08,035 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:56:08,036 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:56:08,036 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:56:08,036 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:08,036 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:56:08,037 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:56:08,037 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:56:08,037 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:08,037 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:56:08,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:08,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:08,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:08,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:08,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:08,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:08,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:08,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:08,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:08,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:08,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:56:08,039 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:56:08,039 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:56:08,039 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:56:08,039 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:56:08,039 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:56:08,039 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:56:08,045 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:08,045 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:08,045 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:08,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:08,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:08,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:08,045 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:56:08,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:08,045 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:08,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:08,046 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:56:08,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:56:08,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:08,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:08,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:08,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:08,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:08,048 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:08,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:08,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:08,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:08,050 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:08,050 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:56:08,050 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:08,050 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:08,050 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:56:08,050 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:56:08,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:56:08,050 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:56:08,051 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:08,051 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:56:08,051 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:56:08,051 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:56:08,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:56:08,051 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:08,051 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:08,051 DEBUG [Thread-3656] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:56:08,051 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:08,051 DEBUG [Thread-3656] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:56:08,058 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/.tmp/A/6271daa9e1ef4f9cac76ac21c645489e as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/A/6271daa9e1ef4f9cac76ac21c645489e 2023-04-25 02:56:08,062 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in d6e38fead90b2a328d380bf5577ff0ce/A of d6e38fead90b2a328d380bf5577ff0ce into 6271daa9e1ef4f9cac76ac21c645489e(size=25.3 K), total size for store is 50.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 02:56:08,062 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for d6e38fead90b2a328d380bf5577ff0ce: 2023-04-25 02:56:08,062 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce., storeName=d6e38fead90b2a328d380bf5577ff0ce/A, priority=11, startTime=1682391367614; duration=0sec 2023-04-25 02:56:08,062 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 02:56:08,151 DEBUG [Thread-3656] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:56:08,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:56:08,154 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:56:08,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:56:08,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:56:08,155 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:56:08,155 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:56:08,155 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:56:08,155 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:56:08,155 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:08,155 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:08,156 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:08,156 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:56:08,157 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:56:08,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:56:08,157 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:08,157 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:56:08,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:08,157 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:08,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:56:08,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:08,158 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:56:08,158 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:56:08,158 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:56:08,158 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:56:08,158 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:56:08,158 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:56:08,158 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:56:08,158 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:56:08,158 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. 2023-04-25 02:56:08,158 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. started... 2023-04-25 02:56:08,159 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d6e38fead90b2a328d380bf5577ff0ce: 2023-04-25 02:56:08,159 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. 2023-04-25 02:56:08,159 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:56:08,159 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:56:08,159 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:08,159 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:56:08,159 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:56:08,160 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:08,160 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:08,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:08,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:08,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:08,161 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:08,161 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:56:08,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:08,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:08,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:08,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:08,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:08,162 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:56:08,162 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@106e79e8[Count = 0] remaining members to acquire global barrier 2023-04-25 02:56:08,162 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:56:08,162 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:08,165 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:08,165 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:08,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:08,165 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:08,165 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:56:08,165 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:56:08,165 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:56:08,165 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:56:08,166 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:56:08,166 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:56:08,166 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:56:08,166 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:08,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:08,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:08,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:08,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:08,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:08,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:08,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:08,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:08,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:08,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:08,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:56:08,169 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:56:08,169 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:56:08,169 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:56:08,169 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:56:08,169 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:56:08,169 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:56:08,170 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:08,171 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:08,171 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:08,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:08,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:08,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:08,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:08,171 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:56:08,172 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:08,172 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:56:08,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:56:08,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:08,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:08,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:08,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:08,174 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:08,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:08,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:08,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:08,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:08,176 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:08,176 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:08,176 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:08,176 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:08,176 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:56:08,176 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:56:08,176 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:56:08,176 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:56:08,176 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:08,176 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:08,176 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:08,176 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:56:08,176 DEBUG [Thread-3656] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:56:08,176 DEBUG [Thread-3656] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:56:08,176 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:56:08,176 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:56:08,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:56:08,177 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:56:08,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:56:08,276 DEBUG [Thread-3656] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:56:08,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:56:08,278 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:56:08,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:56:08,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:56:08,280 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:56:08,280 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:56:08,280 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:56:08,280 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:56:08,280 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:08,280 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:08,283 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:56:08,283 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:08,283 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:56:08,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:56:08,283 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:08,283 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:56:08,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:08,284 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:08,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:56:08,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:08,284 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:56:08,284 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:56:08,285 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:56:08,285 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:56:08,285 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:56:08,285 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:56:08,285 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:56:08,285 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:56:08,285 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. 2023-04-25 02:56:08,285 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. started... 2023-04-25 02:56:08,285 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d6e38fead90b2a328d380bf5577ff0ce: 2023-04-25 02:56:08,285 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. 2023-04-25 02:56:08,285 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:56:08,285 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:56:08,285 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:08,285 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:56:08,285 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:56:08,287 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:08,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:08,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:08,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:08,287 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:08,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:08,287 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:08,287 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:56:08,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:08,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:08,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:08,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:08,289 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:56:08,289 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6539a05d[Count = 0] remaining members to acquire global barrier 2023-04-25 02:56:08,289 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:56:08,289 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:08,290 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:08,290 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:08,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:08,290 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:56:08,290 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:56:08,290 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:56:08,291 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:08,291 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:56:08,291 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:08,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:08,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:08,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:08,291 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:56:08,291 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:56:08,291 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:56:08,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:08,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:08,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:08,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:08,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:08,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:08,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:08,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:56:08,294 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:56:08,294 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:56:08,294 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:56:08,294 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:56:08,294 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:56:08,294 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:56:08,301 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:08,301 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:08,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:08,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:08,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:08,301 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:08,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:08,301 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:56:08,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:08,302 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:08,302 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:56:08,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:56:08,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:08,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:08,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:08,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:08,303 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:08,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:08,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:08,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:08,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:08,306 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:08,306 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:08,306 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:08,306 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:08,306 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:08,306 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:08,306 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:08,306 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:56:08,306 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:56:08,306 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:56:08,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:56:08,306 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:56:08,306 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:56:08,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:56:08,311 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:56:08,311 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:56:08,311 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:56:08,311 DEBUG [Thread-3656] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:56:08,311 DEBUG [Thread-3656] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:56:08,411 DEBUG [Thread-3656] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:56:08,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:56:08,414 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:56:08,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:56:08,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:56:08,415 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:56:08,415 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:56:08,415 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:56:08,415 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:56:08,415 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:08,415 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:08,416 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:56:08,416 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:08,416 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:56:08,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:56:08,417 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:08,417 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:56:08,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:08,417 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:08,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:56:08,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:08,418 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:56:08,418 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:56:08,418 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:56:08,418 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:56:08,418 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:56:08,418 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:56:08,418 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:56:08,418 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:56:08,418 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. 2023-04-25 02:56:08,418 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. started... 2023-04-25 02:56:08,418 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d6e38fead90b2a328d380bf5577ff0ce: 2023-04-25 02:56:08,418 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. 2023-04-25 02:56:08,418 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:56:08,418 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:56:08,418 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:08,419 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:56:08,419 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:56:08,420 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:08,420 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:08,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:08,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:08,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:08,420 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:08,420 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:56:08,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:08,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:08,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:08,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:08,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:08,423 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:56:08,423 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1c7e8a2a[Count = 0] remaining members to acquire global barrier 2023-04-25 02:56:08,423 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:56:08,423 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:08,426 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:08,426 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:08,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:08,426 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:56:08,426 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:56:08,426 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:08,426 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:56:08,426 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:56:08,427 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:56:08,427 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:08,427 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:56:08,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:08,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:08,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:08,427 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:56:08,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:08,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:08,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:08,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:08,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:08,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:08,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:08,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:56:08,430 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:56:08,430 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:56:08,430 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:56:08,430 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:56:08,430 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:56:08,430 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:56:08,432 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:08,432 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:08,432 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:56:08,432 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:08,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:08,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:08,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:08,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:08,432 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:56:08,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:56:08,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:08,433 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:08,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:08,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:08,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:08,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:08,434 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:08,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:08,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:08,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:08,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:08,440 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:08,440 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:08,440 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:08,440 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:56:08,440 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:56:08,440 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:56:08,440 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:56:08,440 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:56:08,440 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:08,440 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:08,440 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:08,440 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:08,440 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:56:08,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:56:08,440 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:56:08,440 DEBUG [Thread-3656] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:56:08,440 DEBUG [Thread-3656] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:56:08,441 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:56:08,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:56:08,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(9158): Flush requested on d6e38fead90b2a328d380bf5577ff0ce 2023-04-25 02:56:08,530 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing d6e38fead90b2a328d380bf5577ff0ce 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-25 02:56:08,531 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6e38fead90b2a328d380bf5577ff0ce, store=A 2023-04-25 02:56:08,531 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:56:08,531 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6e38fead90b2a328d380bf5577ff0ce, store=B 2023-04-25 02:56:08,531 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:56:08,531 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6e38fead90b2a328d380bf5577ff0ce, store=C 2023-04-25 02:56:08,531 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:56:08,541 DEBUG [Thread-3656] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:56:08,555 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:56:08,557 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:56:08,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:56:08,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:56:08,560 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:56:08,560 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:56:08,560 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:56:08,560 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:56:08,560 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:08,561 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:08,562 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:56:08,562 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:56:08,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:56:08,562 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:08,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:08,562 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:08,562 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:56:08,563 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:08,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:56:08,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:08,563 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:56:08,564 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:56:08,564 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:56:08,564 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:56:08,564 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:56:08,565 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:56:08,570 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11854, totalHeapSize=27176, min timestamp=1682391368555, max timestamp=1682391368566Num uniques 859; 2023-04-25 02:56:08,571 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=11854, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-25 02:56:08,573 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:56:08,573 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:56:08,573 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. 2023-04-25 02:56:08,574 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. started... 2023-04-25 02:56:08,574 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. as already flushing 2023-04-25 02:56:08,574 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. 2023-04-25 02:56:08,574 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:56:08,574 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:56:08,574 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:08,574 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:56:08,574 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:56:08,575 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11854, totalHeapSize=27176, min timestamp=1682391368555, max timestamp=1682391368566Num uniques 859; 2023-04-25 02:56:08,575 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=11854, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-25 02:56:08,575 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:08,575 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:08,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:08,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:08,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:08,576 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682391368555, max timestamp=1682391368566Num uniques 859; 2023-04-25 02:56:08,576 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-25 02:56:08,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:08,576 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:08,576 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:56:08,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:08,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:08,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:08,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:08,580 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:56:08,580 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@513d007f[Count = 0] remaining members to acquire global barrier 2023-04-25 02:56:08,580 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:56:08,580 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:08,581 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:08,581 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:08,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:08,581 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:56:08,581 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:56:08,581 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:56:08,581 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:08,581 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:56:08,584 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:08,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:08,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:08,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:08,584 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:56:08,584 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:56:08,584 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:56:08,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:08,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:08,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:08,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:08,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:08,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:08,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:08,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:56:08,586 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:56:08,586 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:56:08,586 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:56:08,586 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:56:08,586 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:56:08,586 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:56:08,588 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:08,588 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:08,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:08,588 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:08,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:08,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:08,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:08,588 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:56:08,588 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:56:08,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:56:08,588 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:08,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:08,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:08,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:08,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:08,590 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:08,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:08,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:08,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:08,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:08,592 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:08,592 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:56:08,592 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:56:08,592 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:56:08,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:56:08,592 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:08,592 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:56:08,592 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:08,592 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:56:08,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39458 deadline: 1682391428588, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:08,592 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:56:08,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:56:08,592 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:56:08,592 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:56:08,592 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:08,593 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:08,593 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:08,593 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:08,593 DEBUG [Thread-3656] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:56:08,593 DEBUG [Thread-3656] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:56:08,593 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:08,593 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:08,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39522 deadline: 1682391428589, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:08,595 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:08,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39482 deadline: 1682391428590, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:08,595 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:08,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39526 deadline: 1682391428591, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:08,596 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:08,597 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39536 deadline: 1682391428592, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:08,696 DEBUG [Thread-3656] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:56:08,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:56:08,699 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:08,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39458 deadline: 1682391428697, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:08,699 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:08,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39522 deadline: 1682391428697, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:08,700 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:08,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39482 deadline: 1682391428697, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:08,700 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:08,700 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:56:08,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39536 deadline: 1682391428698, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:08,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:56:08,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:56:08,701 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:08,701 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:56:08,701 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:56:08,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39526 deadline: 1682391428700, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:08,701 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:56:08,702 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:56:08,704 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:08,704 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:08,706 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:56:08,706 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:56:08,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:56:08,706 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:08,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:08,706 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:08,706 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:56:08,706 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:08,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:56:08,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:08,707 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:56:08,707 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:56:08,707 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:56:08,707 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:56:08,707 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:56:08,707 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:56:08,707 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:56:08,707 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. 2023-04-25 02:56:08,707 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. started... 2023-04-25 02:56:08,707 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. as already flushing 2023-04-25 02:56:08,707 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. 2023-04-25 02:56:08,707 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:56:08,707 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:56:08,707 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:08,707 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:56:08,708 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:56:08,710 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:08,710 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:08,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:08,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:08,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:08,710 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:08,710 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:56:08,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:08,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:08,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:08,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:08,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:08,712 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:56:08,712 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@327f8289[Count = 0] remaining members to acquire global barrier 2023-04-25 02:56:08,712 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:56:08,712 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:08,713 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:08,713 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:08,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:08,713 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:56:08,713 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:56:08,713 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:56:08,713 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:08,713 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:56:08,714 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:08,714 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:56:08,714 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:56:08,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:08,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:08,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:08,714 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:56:08,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:08,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:08,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:08,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:08,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:08,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:08,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:08,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:56:08,717 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:56:08,717 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:56:08,717 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:56:08,717 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:56:08,717 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:56:08,717 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:56:08,718 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:08,718 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:08,718 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:56:08,718 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:08,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:08,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:56:08,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:56:08,719 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:08,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:08,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:08,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:08,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:08,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:08,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:08,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:08,720 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:08,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:08,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:08,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:08,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:08,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:08,723 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:08,724 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:56:08,724 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:08,724 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:56:08,724 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:56:08,724 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:08,724 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:56:08,724 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:56:08,724 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:56:08,724 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:56:08,724 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:08,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:56:08,724 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:08,724 DEBUG [Thread-3656] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:56:08,724 DEBUG [Thread-3656] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:56:08,724 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:08,724 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:08,725 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:56:08,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:56:08,824 DEBUG [Thread-3656] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:56:08,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:56:08,826 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:56:08,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:56:08,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:56:08,828 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:56:08,828 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:56:08,828 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:56:08,828 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:56:08,828 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:08,828 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:08,829 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:56:08,829 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:56:08,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:56:08,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:08,830 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:08,830 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:08,830 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:08,830 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:56:08,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:56:08,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:08,830 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:56:08,831 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:56:08,831 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:56:08,831 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:56:08,831 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:56:08,831 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:56:08,831 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:56:08,831 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:56:08,831 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. 2023-04-25 02:56:08,831 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. started... 2023-04-25 02:56:08,831 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. as already flushing 2023-04-25 02:56:08,831 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. 2023-04-25 02:56:08,831 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:56:08,831 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:56:08,832 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:08,832 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:56:08,832 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:56:08,833 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:08,833 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:08,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:08,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:08,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:08,833 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:08,833 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:56:08,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:08,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:08,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:08,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:08,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:08,835 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:56:08,835 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1a6cf7d3[Count = 0] remaining members to acquire global barrier 2023-04-25 02:56:08,835 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:56:08,835 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:08,836 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:08,836 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:08,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:08,836 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:56:08,836 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:56:08,836 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:56:08,836 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:08,836 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:56:08,838 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:08,838 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:56:08,838 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:56:08,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:08,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:08,838 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:56:08,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:08,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:08,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:08,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:08,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:08,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:08,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:08,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:08,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:56:08,840 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:56:08,840 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:56:08,840 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:56:08,841 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:56:08,841 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:56:08,841 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:56:08,842 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:08,842 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:08,842 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:08,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:08,842 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:56:08,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:08,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:08,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:08,842 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:56:08,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:56:08,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:08,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:08,843 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:08,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:08,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:08,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:08,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:08,844 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:08,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:08,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:08,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:08,854 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:08,854 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:08,854 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:08,854 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:08,854 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:08,854 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:08,854 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:08,854 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:56:08,854 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:56:08,854 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:56:08,854 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:56:08,854 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:56:08,854 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:56:08,854 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:56:08,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:56:08,854 DEBUG [Thread-3656] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:56:08,854 DEBUG [Thread-3656] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:56:08,855 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:56:08,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:56:08,903 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:08,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39522 deadline: 1682391428901, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:08,903 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:08,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39482 deadline: 1682391428901, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:08,904 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:08,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39458 deadline: 1682391428901, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:08,905 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:08,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39536 deadline: 1682391428902, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:08,906 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:08,906 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39526 deadline: 1682391428902, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:08,949 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023042533c167e94fd8441b8294fb6cc250993f to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042533c167e94fd8441b8294fb6cc250993f 2023-04-25 02:56:08,954 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=132, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/.tmp/A/90517b102c6b4392ad8ce64ae33c519f 2023-04-25 02:56:08,954 DEBUG [Thread-3656] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:56:08,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:56:08,957 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:56:08,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:56:08,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:56:08,958 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:56:08,958 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:56:08,958 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:56:08,958 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:56:08,959 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:08,959 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:08,962 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:56:08,962 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:08,963 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:56:08,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:56:08,963 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:08,963 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:56:08,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:08,963 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:08,965 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=132 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/.tmp/B/a57196e0e9ca474bbbc0104d2eeee473 2023-04-25 02:56:08,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:56:08,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:08,965 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:56:08,965 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:56:08,965 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:56:08,965 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:56:08,965 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:56:08,965 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:56:08,965 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:56:08,965 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:56:08,965 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. 2023-04-25 02:56:08,966 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. started... 2023-04-25 02:56:08,966 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. as already flushing 2023-04-25 02:56:08,966 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. 2023-04-25 02:56:08,966 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:56:08,966 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:56:08,966 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:08,966 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:56:08,966 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:56:08,968 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:08,968 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:08,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:08,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:08,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:08,968 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:08,968 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:56:08,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:08,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:08,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:08,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:08,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:08,970 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:56:08,970 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@722b016e[Count = 0] remaining members to acquire global barrier 2023-04-25 02:56:08,970 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:56:08,970 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:08,971 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:08,971 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:08,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:08,971 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:56:08,971 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:56:08,972 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:08,972 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:56:08,972 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:56:08,973 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:56:08,973 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:56:08,973 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:08,973 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:56:08,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:08,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:08,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:08,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:08,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:08,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:08,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:08,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:08,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:08,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:08,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:56:08,977 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:56:08,977 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:56:08,977 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:56:08,977 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:56:08,977 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:56:08,977 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:56:08,982 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:08,983 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:08,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:08,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:08,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:08,983 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:08,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:08,983 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:56:08,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:08,983 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:08,983 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:56:08,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:56:08,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:08,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:08,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:08,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:08,985 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:08,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:08,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:08,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:08,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:08,988 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:08,988 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:08,988 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:08,988 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:08,988 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:08,988 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:08,988 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:08,988 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:56:08,988 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:56:08,988 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:56:08,989 DEBUG [Thread-3656] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:56:08,989 DEBUG [Thread-3656] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:56:08,995 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:56:08,995 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:56:08,995 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:56:08,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:56:08,995 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:56:08,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:56:09,089 DEBUG [Thread-3656] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:56:09,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:56:09,091 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:56:09,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:56:09,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:56:09,092 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:56:09,092 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:56:09,092 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:56:09,092 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:56:09,093 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:09,093 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:09,094 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:56:09,094 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:09,094 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:56:09,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:56:09,094 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:09,095 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:56:09,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:09,095 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:09,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:56:09,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:09,096 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:56:09,096 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:56:09,096 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:56:09,096 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:56:09,096 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:56:09,096 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:56:09,096 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:56:09,096 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:56:09,096 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. 2023-04-25 02:56:09,096 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. started... 2023-04-25 02:56:09,096 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. as already flushing 2023-04-25 02:56:09,096 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. 2023-04-25 02:56:09,096 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:56:09,097 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:56:09,097 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:09,097 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:56:09,097 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:56:09,098 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:09,098 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:09,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:09,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:09,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:09,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:09,098 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:09,099 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:56:09,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:09,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:09,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:09,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:09,100 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:56:09,100 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4b2cc9ac[Count = 0] remaining members to acquire global barrier 2023-04-25 02:56:09,100 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:56:09,100 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:09,102 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:09,102 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:09,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:09,102 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:56:09,102 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:56:09,102 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:56:09,102 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:09,102 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:56:09,104 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:09,104 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:56:09,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:09,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:09,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:09,104 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:56:09,104 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:56:09,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:09,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:09,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:09,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:09,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:09,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:09,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:09,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:56:09,106 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:56:09,106 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:56:09,106 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:56:09,106 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:56:09,106 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:56:09,106 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:56:09,108 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:09,108 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:56:09,108 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:09,108 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:09,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:09,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:09,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:09,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:09,108 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:56:09,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:56:09,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:09,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:09,109 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:09,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:09,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:09,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:09,110 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:09,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:09,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:09,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:09,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:09,112 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:09,112 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:09,112 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:09,112 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:56:09,112 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:56:09,112 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:56:09,112 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:56:09,112 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:56:09,112 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:09,112 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:09,112 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:09,112 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:09,112 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:56:09,112 DEBUG [Thread-3656] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:56:09,112 DEBUG [Thread-3656] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:56:09,112 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:56:09,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:56:09,113 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:56:09,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:56:09,206 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:09,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39482 deadline: 1682391429204, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:09,207 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:09,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39522 deadline: 1682391429205, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:09,209 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:09,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39458 deadline: 1682391429206, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:09,210 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:09,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39536 deadline: 1682391429206, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:09,211 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:09,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39526 deadline: 1682391429208, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:09,213 DEBUG [Thread-3656] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:56:09,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:56:09,215 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:56:09,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:56:09,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:56:09,216 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:56:09,216 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:56:09,216 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:56:09,216 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:56:09,216 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:09,216 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:09,218 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:56:09,218 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:09,218 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:56:09,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:56:09,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:09,219 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:09,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:56:09,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:09,220 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:09,220 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:56:09,220 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:56:09,220 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:56:09,220 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:56:09,220 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:56:09,220 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:56:09,220 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:56:09,221 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:56:09,221 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:56:09,221 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. 2023-04-25 02:56:09,221 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. started... 2023-04-25 02:56:09,221 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. as already flushing 2023-04-25 02:56:09,221 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. 2023-04-25 02:56:09,221 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:56:09,221 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:56:09,221 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:09,221 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:56:09,221 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:56:09,222 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:09,222 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:09,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:09,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:09,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:09,223 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:09,223 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:56:09,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:09,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:09,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:09,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:09,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:09,224 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:56:09,224 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@589618a[Count = 0] remaining members to acquire global barrier 2023-04-25 02:56:09,224 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:56:09,224 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:09,225 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:09,226 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:09,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:09,226 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:56:09,226 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:56:09,226 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:56:09,226 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:09,226 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:56:09,227 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:09,227 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:56:09,227 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:56:09,227 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:56:09,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:09,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:09,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:09,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:09,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:09,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:09,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:09,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:09,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:09,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:09,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:56:09,230 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:56:09,230 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:56:09,230 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:56:09,230 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:56:09,230 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:56:09,230 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:56:09,232 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:09,232 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:09,232 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:09,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:09,232 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:56:09,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:09,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:09,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:09,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:09,233 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:09,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:09,233 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:56:09,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:56:09,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:09,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:09,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:09,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:09,234 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:09,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:09,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:09,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:09,236 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:56:09,236 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:56:09,236 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:56:09,236 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:09,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:56:09,237 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:56:09,237 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:56:09,237 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:56:09,236 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:09,237 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:09,237 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:09,237 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:09,237 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:09,237 DEBUG [Thread-3656] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:56:09,237 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:56:09,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:56:09,237 DEBUG [Thread-3656] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:56:09,237 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:09,337 DEBUG [Thread-3656] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:56:09,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:56:09,339 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:56:09,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:56:09,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:56:09,341 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:56:09,341 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:56:09,341 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:56:09,341 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:56:09,341 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:09,341 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:09,342 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:56:09,342 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:09,343 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:56:09,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:56:09,343 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:09,343 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:56:09,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:09,344 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:09,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:56:09,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:09,344 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:56:09,345 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:56:09,345 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:56:09,345 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:56:09,345 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:56:09,345 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:56:09,345 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:56:09,345 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:56:09,345 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. 2023-04-25 02:56:09,345 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. started... 2023-04-25 02:56:09,345 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. as already flushing 2023-04-25 02:56:09,345 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. 2023-04-25 02:56:09,345 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:56:09,345 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:56:09,345 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:09,346 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:56:09,346 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:56:09,351 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:09,351 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:09,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:09,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:09,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:09,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:09,352 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:09,352 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:56:09,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:09,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:09,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:09,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:09,354 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:56:09,354 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7826cc87[Count = 0] remaining members to acquire global barrier 2023-04-25 02:56:09,354 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:56:09,354 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:09,355 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:09,355 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:09,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:09,355 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:09,355 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:56:09,356 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:56:09,356 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:56:09,356 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:56:09,358 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:09,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:09,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:09,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:09,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:09,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:09,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:09,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:09,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:09,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:09,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:09,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:56:09,360 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:56:09,360 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:56:09,360 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:56:09,360 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:56:09,360 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:56:09,360 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:56:09,360 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:56:09,360 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:56:09,361 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:56:09,362 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:09,362 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:09,362 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:09,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:09,362 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:56:09,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:09,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:09,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:09,363 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:56:09,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:56:09,363 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:09,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:09,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:09,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:09,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:09,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:09,364 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:09,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:09,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:09,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:09,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:09,367 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:09,367 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:09,367 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:09,367 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:09,367 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:56:09,367 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:09,368 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:09,368 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:56:09,368 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:56:09,368 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:56:09,368 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:56:09,368 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:56:09,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:56:09,368 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:56:09,368 DEBUG [Thread-3656] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:56:09,368 DEBUG [Thread-3656] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:56:09,368 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:09,369 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:56:09,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:56:09,375 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=132 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/.tmp/C/869045a69f634dd6886f7f7c124e49a4 2023-04-25 02:56:09,379 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/.tmp/A/90517b102c6b4392ad8ce64ae33c519f as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/A/90517b102c6b4392ad8ce64ae33c519f 2023-04-25 02:56:09,385 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/A/90517b102c6b4392ad8ce64ae33c519f, entries=300, sequenceid=132, filesize=45.7 K 2023-04-25 02:56:09,386 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/.tmp/B/a57196e0e9ca474bbbc0104d2eeee473 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/B/a57196e0e9ca474bbbc0104d2eeee473 2023-04-25 02:56:09,389 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/B/a57196e0e9ca474bbbc0104d2eeee473, entries=100, sequenceid=132, filesize=9.4 K 2023-04-25 02:56:09,390 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/.tmp/C/869045a69f634dd6886f7f7c124e49a4 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/C/869045a69f634dd6886f7f7c124e49a4 2023-04-25 02:56:09,393 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/C/869045a69f634dd6886f7f7c124e49a4, entries=100, sequenceid=132, filesize=9.4 K 2023-04-25 02:56:09,394 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for d6e38fead90b2a328d380bf5577ff0ce in 864ms, sequenceid=132, compaction requested=true 2023-04-25 02:56:09,394 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for d6e38fead90b2a328d380bf5577ff0ce: 2023-04-25 02:56:09,394 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 02:56:09,394 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-25 02:56:09,394 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-25 02:56:09,395 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 02:56:09,395 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 02:56:09,395 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 98430 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-25 02:56:09,396 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 33586 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-25 02:56:09,396 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HStore(1912): d6e38fead90b2a328d380bf5577ff0ce/A is initiating minor compaction (all files) 2023-04-25 02:56:09,396 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HStore(1912): d6e38fead90b2a328d380bf5577ff0ce/B is initiating minor compaction (all files) 2023-04-25 02:56:09,396 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of d6e38fead90b2a328d380bf5577ff0ce/A in TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. 2023-04-25 02:56:09,396 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d6e38fead90b2a328d380bf5577ff0ce/B in TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. 2023-04-25 02:56:09,396 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/A/6271daa9e1ef4f9cac76ac21c645489e, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/A/6d439419060d4cd4a5bb0300afe993c6, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/A/90517b102c6b4392ad8ce64ae33c519f] into tmpdir=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/.tmp, totalSize=96.1 K 2023-04-25 02:56:09,396 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/B/a0d9a6ee1308466f8489abd8adc7ded5, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/B/2ea6f369063149b38f4755ea9d2aaa4a, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/B/a57196e0e9ca474bbbc0104d2eeee473] into tmpdir=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/.tmp, totalSize=32.8 K 2023-04-25 02:56:09,396 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting 6271daa9e1ef4f9cac76ac21c645489e, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=81, earliestPutTs=1682391364428 2023-04-25 02:56:09,396 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting a0d9a6ee1308466f8489abd8adc7ded5, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=81, earliestPutTs=1682391364428 2023-04-25 02:56:09,397 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting 6d439419060d4cd4a5bb0300afe993c6, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=115, earliestPutTs=1682391366369 2023-04-25 02:56:09,397 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting 2ea6f369063149b38f4755ea9d2aaa4a, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=115, earliestPutTs=1682391366369 2023-04-25 02:56:09,397 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting 90517b102c6b4392ad8ce64ae33c519f, keycount=300, bloomtype=ROW, size=45.7 K, encoding=NONE, compression=NONE, seqNum=132, earliestPutTs=1682391368515 2023-04-25 02:56:09,397 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting a57196e0e9ca474bbbc0104d2eeee473, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=132, earliestPutTs=1682391368528 2023-04-25 02:56:09,404 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] throttle.PressureAwareThroughputController(145): d6e38fead90b2a328d380bf5577ff0ce#B#compaction#508 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 53.85 MB/second 2023-04-25 02:56:09,408 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] throttle.PressureAwareThroughputController(145): d6e38fead90b2a328d380bf5577ff0ce#A#compaction#507 average throughput is 3.94 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-25 02:56:09,468 DEBUG [Thread-3656] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:56:09,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:56:09,470 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:56:09,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:56:09,472 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:56:09,472 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:56:09,472 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:56:09,472 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:56:09,472 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:56:09,472 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:09,472 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:09,473 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:56:09,473 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:09,473 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:56:09,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:56:09,474 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:09,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:09,474 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:56:09,474 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:09,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:56:09,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:09,475 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:56:09,475 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:56:09,475 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:56:09,475 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:56:09,475 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:56:09,475 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:56:09,475 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:56:09,475 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:56:09,475 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. 2023-04-25 02:56:09,475 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. started... 2023-04-25 02:56:09,475 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d6e38fead90b2a328d380bf5577ff0ce 3/3 column families, dataSize=127.47 KB heapSize=355.69 KB 2023-04-25 02:56:09,476 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6e38fead90b2a328d380bf5577ff0ce, store=A 2023-04-25 02:56:09,476 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 02:56:09,476 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6e38fead90b2a328d380bf5577ff0ce, store=B 2023-04-25 02:56:09,476 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 02:56:09,476 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6e38fead90b2a328d380bf5577ff0ce, store=C 2023-04-25 02:56:09,476 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 02:56:09,492 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304251ac3436ad08d4c2ebfa525ee015b5abd to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304251ac3436ad08d4c2ebfa525ee015b5abd 2023-04-25 02:56:09,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(9158): Flush requested on d6e38fead90b2a328d380bf5577ff0ce 2023-04-25 02:56:09,710 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. as already flushing 2023-04-25 02:56:09,720 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:09,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39458 deadline: 1682391429715, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:09,721 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:09,721 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:09,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39526 deadline: 1682391429716, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:09,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39536 deadline: 1682391429715, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:09,721 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:09,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39482 deadline: 1682391429718, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:09,723 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:09,723 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39522 deadline: 1682391429721, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:09,818 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/.tmp/B/a15f1823d59b4c43b1e56730d7903285 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/B/a15f1823d59b4c43b1e56730d7903285 2023-04-25 02:56:09,822 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in d6e38fead90b2a328d380bf5577ff0ce/B of d6e38fead90b2a328d380bf5577ff0ce into a15f1823d59b4c43b1e56730d7903285(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 02:56:09,823 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d6e38fead90b2a328d380bf5577ff0ce: 2023-04-25 02:56:09,823 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce., storeName=d6e38fead90b2a328d380bf5577ff0ce/B, priority=13, startTime=1682391369394; duration=0sec 2023-04-25 02:56:09,823 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 02:56:09,823 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-25 02:56:09,823 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:09,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39458 deadline: 1682391429822, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:09,824 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:09,824 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 33586 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-25 02:56:09,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39526 deadline: 1682391429822, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:09,824 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HStore(1912): d6e38fead90b2a328d380bf5577ff0ce/C is initiating minor compaction (all files) 2023-04-25 02:56:09,824 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d6e38fead90b2a328d380bf5577ff0ce/C in TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. 2023-04-25 02:56:09,824 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:09,824 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/C/bc39437696f2440cba5221a588d9b824, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/C/ac631873d4384b54a0d9595795e44b24, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/C/869045a69f634dd6886f7f7c124e49a4] into tmpdir=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/.tmp, totalSize=32.8 K 2023-04-25 02:56:09,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39482 deadline: 1682391429822, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:09,825 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:09,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39536 deadline: 1682391429822, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:09,825 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting bc39437696f2440cba5221a588d9b824, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=81, earliestPutTs=1682391364428 2023-04-25 02:56:09,825 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting ac631873d4384b54a0d9595795e44b24, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=115, earliestPutTs=1682391366369 2023-04-25 02:56:09,825 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting 869045a69f634dd6886f7f7c124e49a4, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=132, earliestPutTs=1682391368528 2023-04-25 02:56:09,827 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:09,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39522 deadline: 1682391429824, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:09,831 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] throttle.PressureAwareThroughputController(145): d6e38fead90b2a328d380bf5577ff0ce#C#compaction#510 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-25 02:56:09,841 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/.tmp/C/a2b51c3bd3dd4888a104f7a28abe1b2f as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/C/a2b51c3bd3dd4888a104f7a28abe1b2f 2023-04-25 02:56:09,841 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/.tmp/A/da5081f299cd46f2a031f22dda953627 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/A/da5081f299cd46f2a031f22dda953627 2023-04-25 02:56:09,847 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in d6e38fead90b2a328d380bf5577ff0ce/A of d6e38fead90b2a328d380bf5577ff0ce into da5081f299cd46f2a031f22dda953627(size=25.5 K), total size for store is 25.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 02:56:09,847 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for d6e38fead90b2a328d380bf5577ff0ce: 2023-04-25 02:56:09,847 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce., storeName=d6e38fead90b2a328d380bf5577ff0ce/A, priority=13, startTime=1682391369394; duration=0sec 2023-04-25 02:56:09,847 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 02:56:09,847 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in d6e38fead90b2a328d380bf5577ff0ce/C of d6e38fead90b2a328d380bf5577ff0ce into a2b51c3bd3dd4888a104f7a28abe1b2f(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 02:56:09,847 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d6e38fead90b2a328d380bf5577ff0ce: 2023-04-25 02:56:09,847 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce., storeName=d6e38fead90b2a328d380bf5577ff0ce/C, priority=13, startTime=1682391369395; duration=0sec 2023-04-25 02:56:09,848 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 02:56:09,898 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=166, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/.tmp/A/04ed9120bfe542a19ee411e7e2d31eb8 2023-04-25 02:56:09,907 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=166 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/.tmp/B/9acecb7ac5eb44e18e76902aff7f6b28 2023-04-25 02:56:09,923 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=166 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/.tmp/C/5995fd86db7f4e3ea56b0f11f787877e 2023-04-25 02:56:09,927 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/.tmp/A/04ed9120bfe542a19ee411e7e2d31eb8 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/A/04ed9120bfe542a19ee411e7e2d31eb8 2023-04-25 02:56:09,931 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/A/04ed9120bfe542a19ee411e7e2d31eb8, entries=150, sequenceid=166, filesize=25.3 K 2023-04-25 02:56:09,931 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/.tmp/B/9acecb7ac5eb44e18e76902aff7f6b28 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/B/9acecb7ac5eb44e18e76902aff7f6b28 2023-04-25 02:56:09,935 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/B/9acecb7ac5eb44e18e76902aff7f6b28, entries=150, sequenceid=166, filesize=11.8 K 2023-04-25 02:56:09,936 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/.tmp/C/5995fd86db7f4e3ea56b0f11f787877e as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/C/5995fd86db7f4e3ea56b0f11f787877e 2023-04-25 02:56:09,939 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/C/5995fd86db7f4e3ea56b0f11f787877e, entries=150, sequenceid=166, filesize=11.8 K 2023-04-25 02:56:09,939 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~355.64 KB/364176, currentSize=53.67 KB/54960 for d6e38fead90b2a328d380bf5577ff0ce in 464ms, sequenceid=166, compaction requested=false 2023-04-25 02:56:09,939 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d6e38fead90b2a328d380bf5577ff0ce: 2023-04-25 02:56:09,939 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. 2023-04-25 02:56:09,939 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:56:09,939 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:56:09,939 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:09,940 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:56:09,940 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:56:09,944 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:09,944 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:09,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:09,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:09,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:09,945 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:09,945 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:56:09,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:09,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:09,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:09,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:09,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:09,946 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:56:09,946 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@522c829c[Count = 0] remaining members to acquire global barrier 2023-04-25 02:56:09,946 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:56:09,947 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:09,948 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:09,948 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:56:09,948 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:09,948 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:09,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:09,948 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:56:09,948 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:56:09,948 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:56:09,951 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:09,951 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:56:09,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:09,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:09,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:09,951 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:56:09,951 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:56:09,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:09,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:09,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:09,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:09,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:09,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:09,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:09,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:56:09,954 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:56:09,954 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:56:09,954 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:56:09,954 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:56:09,954 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:56:09,954 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:56:09,956 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:09,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:09,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:09,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:09,956 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:09,956 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:56:09,956 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:09,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:09,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:09,956 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:56:09,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:56:09,956 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:09,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:09,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:09,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:09,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:09,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:09,958 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:09,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:09,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:09,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:09,960 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:56:09,960 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:56:09,960 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:56:09,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:56:09,960 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:09,961 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:09,961 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:09,961 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:09,961 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:09,961 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:09,961 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:09,961 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:56:09,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:56:09,961 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:56:09,961 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:56:09,961 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:56:09,961 DEBUG [Thread-3656] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:56:09,961 DEBUG [Thread-3656] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:56:10,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(9158): Flush requested on d6e38fead90b2a328d380bf5577ff0ce 2023-04-25 02:56:10,027 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing d6e38fead90b2a328d380bf5577ff0ce 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-25 02:56:10,028 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6e38fead90b2a328d380bf5577ff0ce, store=A 2023-04-25 02:56:10,028 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:56:10,028 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6e38fead90b2a328d380bf5577ff0ce, store=B 2023-04-25 02:56:10,028 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:56:10,028 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6e38fead90b2a328d380bf5577ff0ce, store=C 2023-04-25 02:56:10,028 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:56:10,048 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:10,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39482 deadline: 1682391430044, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:10,049 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:10,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39536 deadline: 1682391430046, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:10,051 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:10,051 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39522 deadline: 1682391430047, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:10,051 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:10,051 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39458 deadline: 1682391430048, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:10,051 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:10,051 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39526 deadline: 1682391430048, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:10,061 DEBUG [Thread-3656] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:56:10,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:56:10,063 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:56:10,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:56:10,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:56:10,064 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:56:10,064 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:56:10,064 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:56:10,064 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:56:10,065 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:10,065 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:10,066 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:10,066 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:56:10,066 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:56:10,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:56:10,067 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:10,067 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:56:10,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:10,067 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:10,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:56:10,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:10,067 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:56:10,068 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:56:10,068 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:56:10,068 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:56:10,068 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:56:10,068 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:56:10,068 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:56:10,068 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:56:10,068 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. 2023-04-25 02:56:10,068 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. started... 2023-04-25 02:56:10,068 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. as already flushing 2023-04-25 02:56:10,068 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. 2023-04-25 02:56:10,068 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:56:10,068 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:56:10,069 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:10,069 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:56:10,069 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:56:10,071 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:10,071 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:10,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:10,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:10,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:10,071 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:10,072 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:56:10,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:10,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:10,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:10,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:10,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:10,073 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:56:10,073 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7f6050dc[Count = 0] remaining members to acquire global barrier 2023-04-25 02:56:10,073 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:56:10,074 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:10,074 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:10,075 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:10,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:10,075 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:56:10,075 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:56:10,075 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:56:10,075 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:10,075 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:56:10,076 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:10,076 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:56:10,076 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:56:10,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:10,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:10,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:10,076 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:56:10,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:10,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:10,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:10,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:10,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:10,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:10,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:10,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:56:10,079 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:56:10,079 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:56:10,079 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:56:10,079 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:56:10,079 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:56:10,079 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:56:10,081 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:10,081 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:56:10,081 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:10,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:10,081 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:10,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:10,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:10,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:10,081 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:56:10,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:56:10,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:10,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:10,082 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:10,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:10,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:10,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:10,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:10,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:10,084 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:10,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:10,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:10,088 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:10,088 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:10,088 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:56:10,088 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:10,088 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:10,088 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:56:10,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:56:10,088 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:56:10,088 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:56:10,088 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:56:10,088 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:56:10,088 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:10,088 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:10,088 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:10,088 DEBUG [Thread-3656] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:56:10,088 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:56:10,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:56:10,088 DEBUG [Thread-3656] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:56:10,151 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:10,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39482 deadline: 1682391430149, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:10,151 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:10,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39536 deadline: 1682391430150, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:10,154 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:10,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39522 deadline: 1682391430152, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:10,154 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:10,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39458 deadline: 1682391430152, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:10,155 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:10,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39526 deadline: 1682391430152, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:10,189 DEBUG [Thread-3656] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:56:10,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:56:10,191 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:56:10,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:56:10,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:56:10,192 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:56:10,192 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:56:10,192 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:56:10,192 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:56:10,192 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:10,193 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:10,194 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:56:10,194 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:56:10,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:56:10,194 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:10,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:10,194 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:10,194 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:56:10,194 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:10,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:56:10,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:10,195 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:56:10,195 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:56:10,195 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:56:10,195 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:56:10,196 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:56:10,196 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:56:10,196 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:56:10,196 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:56:10,196 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. 2023-04-25 02:56:10,196 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. started... 2023-04-25 02:56:10,196 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. as already flushing 2023-04-25 02:56:10,196 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. 2023-04-25 02:56:10,196 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:56:10,196 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:56:10,196 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:10,196 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:56:10,196 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:56:10,197 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:10,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:10,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:10,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:10,197 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:10,198 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:10,198 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:56:10,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:10,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:10,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:10,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:10,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:10,199 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:56:10,199 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@312bfb8b[Count = 0] remaining members to acquire global barrier 2023-04-25 02:56:10,199 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:56:10,199 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:10,208 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:10,208 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:10,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:10,208 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:56:10,208 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:56:10,208 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:56:10,208 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:10,209 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:56:10,210 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:10,210 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:56:10,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:10,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:10,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:10,210 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:56:10,210 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:56:10,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:10,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:10,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:10,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:10,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:10,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:10,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:10,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:56:10,213 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:56:10,213 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:56:10,213 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:56:10,213 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:56:10,213 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:56:10,213 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:56:10,214 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:10,214 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:10,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:10,214 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:10,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:10,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:10,214 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:56:10,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:10,215 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:56:10,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:56:10,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:10,215 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:10,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:10,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:10,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:10,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:10,216 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:10,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:10,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:10,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:10,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:10,218 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:56:10,218 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:56:10,218 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:56:10,218 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:56:10,218 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:10,218 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:10,218 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:10,219 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:10,219 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:10,219 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:10,219 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:10,218 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:56:10,218 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:56:10,219 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:56:10,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:56:10,219 DEBUG [Thread-3656] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:56:10,219 DEBUG [Thread-3656] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:56:10,219 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:56:10,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:56:10,319 DEBUG [Thread-3656] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:56:10,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:56:10,321 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:56:10,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:56:10,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:56:10,323 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:56:10,323 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:56:10,323 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:56:10,323 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:56:10,323 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:10,323 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:10,326 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:10,326 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:56:10,326 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:56:10,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:56:10,326 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:10,326 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:56:10,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:10,327 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:10,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:56:10,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:10,327 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:56:10,327 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:56:10,327 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:56:10,327 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:56:10,327 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:56:10,327 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:56:10,327 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:56:10,327 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:56:10,327 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. 2023-04-25 02:56:10,327 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. started... 2023-04-25 02:56:10,328 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. as already flushing 2023-04-25 02:56:10,328 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. 2023-04-25 02:56:10,328 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:56:10,328 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:56:10,328 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:10,328 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:56:10,328 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:56:10,329 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:10,329 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:10,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:10,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:10,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:10,329 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:10,329 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:56:10,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:10,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:10,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:10,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:10,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:10,331 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:56:10,331 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7fa74bdd[Count = 0] remaining members to acquire global barrier 2023-04-25 02:56:10,331 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:56:10,331 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:10,332 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:10,332 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:10,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:10,332 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:56:10,332 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:56:10,332 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:56:10,332 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:10,332 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:56:10,333 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:10,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:10,333 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:56:10,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:10,333 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:56:10,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:10,334 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:56:10,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:10,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:10,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:10,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:10,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:10,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:10,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:10,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:56:10,336 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:56:10,336 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:56:10,336 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:56:10,336 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:56:10,336 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:56:10,336 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:56:10,339 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:10,339 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:56:10,339 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:10,339 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:10,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:10,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:10,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:10,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:10,339 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:56:10,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:56:10,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:10,340 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:10,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:10,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:10,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:10,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:10,341 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:10,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:10,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:10,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:10,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:10,343 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:10,343 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:56:10,343 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:10,343 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:56:10,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:56:10,343 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:56:10,343 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:56:10,343 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:56:10,343 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:10,343 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:10,343 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:10,343 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:56:10,343 DEBUG [Thread-3656] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:56:10,344 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:56:10,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:56:10,344 DEBUG [Thread-3656] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:56:10,343 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:10,343 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:56:10,344 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:10,353 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:10,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39482 deadline: 1682391430352, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:10,355 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:10,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39536 deadline: 1682391430353, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:10,356 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:10,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39458 deadline: 1682391430355, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:10,357 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:10,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39522 deadline: 1682391430355, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:10,359 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:10,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39526 deadline: 1682391430357, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:10,439 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304258eb646c5e0cd46a5bde4dd33a3eabc36 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304258eb646c5e0cd46a5bde4dd33a3eabc36 2023-04-25 02:56:10,444 DEBUG [Thread-3656] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:56:10,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:56:10,445 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=182, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/.tmp/A/a1680fd4f98842418622f381b1e79ac4 2023-04-25 02:56:10,449 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:56:10,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:56:10,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:56:10,451 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:56:10,451 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:56:10,451 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:56:10,451 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:56:10,453 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:10,453 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:10,454 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:10,454 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:56:10,454 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:56:10,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:56:10,454 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:10,454 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:56:10,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:10,455 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:10,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:56:10,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:10,455 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:56:10,455 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:56:10,455 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:56:10,456 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:56:10,456 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:56:10,456 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:56:10,460 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:56:10,460 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:56:10,460 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. 2023-04-25 02:56:10,460 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. started... 2023-04-25 02:56:10,460 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. as already flushing 2023-04-25 02:56:10,460 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. 2023-04-25 02:56:10,460 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=182 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/.tmp/B/c8e03afd6945439889117a5417535654 2023-04-25 02:56:10,460 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:56:10,460 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:56:10,460 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:10,460 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:56:10,460 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:56:10,462 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:10,462 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:10,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:10,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:10,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:10,462 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:10,462 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:56:10,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:10,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:10,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:10,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:10,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:10,466 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:56:10,466 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@49290ed7[Count = 0] remaining members to acquire global barrier 2023-04-25 02:56:10,467 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:56:10,467 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:10,469 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:10,469 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:10,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:10,469 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:56:10,469 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:56:10,469 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:56:10,470 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:10,470 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:56:10,471 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:10,471 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:56:10,471 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:56:10,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:10,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:10,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:10,471 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=182 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/.tmp/C/1f8032eb88e94d8eb1bb42318f4fdd75 2023-04-25 02:56:10,471 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:56:10,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:10,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:10,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:10,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:10,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:10,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:10,476 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/.tmp/A/a1680fd4f98842418622f381b1e79ac4 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/A/a1680fd4f98842418622f381b1e79ac4 2023-04-25 02:56:10,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:10,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:56:10,476 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:56:10,476 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:56:10,476 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:56:10,477 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:56:10,477 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:56:10,477 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:56:10,479 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:10,479 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:56:10,479 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:10,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:10,479 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:10,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:10,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:10,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:10,479 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/A/a1680fd4f98842418622f381b1e79ac4, entries=200, sequenceid=182, filesize=32.1 K 2023-04-25 02:56:10,480 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:56:10,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:56:10,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:10,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:10,480 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/.tmp/B/c8e03afd6945439889117a5417535654 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/B/c8e03afd6945439889117a5417535654 2023-04-25 02:56:10,480 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:10,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:10,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:10,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:10,482 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:10,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:10,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:10,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:10,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:10,484 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:56:10,484 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:56:10,484 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:56:10,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:56:10,484 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:10,484 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:56:10,484 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:56:10,485 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:56:10,485 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:56:10,484 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:56:10,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:56:10,484 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:10,485 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:10,485 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:10,485 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:10,485 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:10,485 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:10,492 DEBUG [Thread-3656] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:56:10,492 DEBUG [Thread-3656] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:56:10,492 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/B/c8e03afd6945439889117a5417535654, entries=150, sequenceid=182, filesize=11.8 K 2023-04-25 02:56:10,493 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/.tmp/C/1f8032eb88e94d8eb1bb42318f4fdd75 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/C/1f8032eb88e94d8eb1bb42318f4fdd75 2023-04-25 02:56:10,495 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/C/1f8032eb88e94d8eb1bb42318f4fdd75, entries=150, sequenceid=182, filesize=11.8 K 2023-04-25 02:56:10,496 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=114.05 KB/116790 for d6e38fead90b2a328d380bf5577ff0ce in 469ms, sequenceid=182, compaction requested=true 2023-04-25 02:56:10,496 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for d6e38fead90b2a328d380bf5577ff0ce: 2023-04-25 02:56:10,496 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 02:56:10,496 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 02:56:10,496 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-25 02:56:10,496 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-25 02:56:10,496 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 02:56:10,498 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36332 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-25 02:56:10,498 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 84902 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-25 02:56:10,498 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HStore(1912): d6e38fead90b2a328d380bf5577ff0ce/B is initiating minor compaction (all files) 2023-04-25 02:56:10,498 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HStore(1912): d6e38fead90b2a328d380bf5577ff0ce/A is initiating minor compaction (all files) 2023-04-25 02:56:10,498 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d6e38fead90b2a328d380bf5577ff0ce/B in TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. 2023-04-25 02:56:10,498 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of d6e38fead90b2a328d380bf5577ff0ce/A in TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. 2023-04-25 02:56:10,498 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/B/a15f1823d59b4c43b1e56730d7903285, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/B/9acecb7ac5eb44e18e76902aff7f6b28, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/B/c8e03afd6945439889117a5417535654] into tmpdir=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/.tmp, totalSize=35.5 K 2023-04-25 02:56:10,498 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/A/da5081f299cd46f2a031f22dda953627, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/A/04ed9120bfe542a19ee411e7e2d31eb8, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/A/a1680fd4f98842418622f381b1e79ac4] into tmpdir=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/.tmp, totalSize=82.9 K 2023-04-25 02:56:10,499 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting a15f1823d59b4c43b1e56730d7903285, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=132, earliestPutTs=1682391366374 2023-04-25 02:56:10,499 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting da5081f299cd46f2a031f22dda953627, keycount=150, bloomtype=ROW, size=25.5 K, encoding=NONE, compression=NONE, seqNum=132, earliestPutTs=1682391366374 2023-04-25 02:56:10,499 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting 9acecb7ac5eb44e18e76902aff7f6b28, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=166, earliestPutTs=1682391368589 2023-04-25 02:56:10,499 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting 04ed9120bfe542a19ee411e7e2d31eb8, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=166, earliestPutTs=1682391368589 2023-04-25 02:56:10,499 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting c8e03afd6945439889117a5417535654, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=182, earliestPutTs=1682391369719 2023-04-25 02:56:10,500 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting a1680fd4f98842418622f381b1e79ac4, keycount=200, bloomtype=ROW, size=32.1 K, encoding=NONE, compression=NONE, seqNum=182, earliestPutTs=1682391369714 2023-04-25 02:56:10,506 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] throttle.PressureAwareThroughputController(145): d6e38fead90b2a328d380bf5577ff0ce#B#compaction#516 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 53.85 MB/second 2023-04-25 02:56:10,521 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] throttle.PressureAwareThroughputController(145): d6e38fead90b2a328d380bf5577ff0ce#A#compaction#517 average throughput is 1.31 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-25 02:56:10,525 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/.tmp/B/7cec68bee0994e788f89f2e393d4fd6d as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/B/7cec68bee0994e788f89f2e393d4fd6d 2023-04-25 02:56:10,530 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in d6e38fead90b2a328d380bf5577ff0ce/B of d6e38fead90b2a328d380bf5577ff0ce into 7cec68bee0994e788f89f2e393d4fd6d(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 02:56:10,530 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d6e38fead90b2a328d380bf5577ff0ce: 2023-04-25 02:56:10,530 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce., storeName=d6e38fead90b2a328d380bf5577ff0ce/B, priority=13, startTime=1682391370496; duration=0sec 2023-04-25 02:56:10,530 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 02:56:10,530 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-25 02:56:10,531 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36332 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-25 02:56:10,531 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HStore(1912): d6e38fead90b2a328d380bf5577ff0ce/C is initiating minor compaction (all files) 2023-04-25 02:56:10,531 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d6e38fead90b2a328d380bf5577ff0ce/C in TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. 2023-04-25 02:56:10,531 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/C/a2b51c3bd3dd4888a104f7a28abe1b2f, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/C/5995fd86db7f4e3ea56b0f11f787877e, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/C/1f8032eb88e94d8eb1bb42318f4fdd75] into tmpdir=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/.tmp, totalSize=35.5 K 2023-04-25 02:56:10,531 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting a2b51c3bd3dd4888a104f7a28abe1b2f, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=132, earliestPutTs=1682391366374 2023-04-25 02:56:10,532 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting 5995fd86db7f4e3ea56b0f11f787877e, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=166, earliestPutTs=1682391368589 2023-04-25 02:56:10,532 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting 1f8032eb88e94d8eb1bb42318f4fdd75, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=182, earliestPutTs=1682391369719 2023-04-25 02:56:10,540 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] throttle.PressureAwareThroughputController(145): d6e38fead90b2a328d380bf5577ff0ce#C#compaction#518 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-25 02:56:10,592 DEBUG [Thread-3656] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:56:10,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:56:10,594 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:56:10,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:56:10,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:56:10,595 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:56:10,595 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:56:10,595 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:56:10,595 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:56:10,596 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:10,596 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:10,600 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:56:10,600 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:10,600 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:56:10,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:56:10,600 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:10,600 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:56:10,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:10,601 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:10,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:56:10,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:10,601 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:56:10,601 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:56:10,601 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:56:10,601 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:56:10,602 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:56:10,602 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:56:10,602 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:56:10,602 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:56:10,602 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. 2023-04-25 02:56:10,602 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. started... 2023-04-25 02:56:10,602 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d6e38fead90b2a328d380bf5577ff0ce 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-25 02:56:10,602 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6e38fead90b2a328d380bf5577ff0ce, store=A 2023-04-25 02:56:10,602 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:56:10,602 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6e38fead90b2a328d380bf5577ff0ce, store=B 2023-04-25 02:56:10,602 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:56:10,602 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6e38fead90b2a328d380bf5577ff0ce, store=C 2023-04-25 02:56:10,602 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:56:10,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(9158): Flush requested on d6e38fead90b2a328d380bf5577ff0ce 2023-04-25 02:56:10,657 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. as already flushing 2023-04-25 02:56:10,665 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:10,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39522 deadline: 1682391430661, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:10,666 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:10,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39482 deadline: 1682391430663, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:10,666 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:10,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39526 deadline: 1682391430664, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:10,668 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:10,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39458 deadline: 1682391430665, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:10,668 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:10,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39536 deadline: 1682391430665, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:10,768 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:10,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39522 deadline: 1682391430766, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:10,769 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:10,769 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39526 deadline: 1682391430768, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:10,769 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:10,769 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39482 deadline: 1682391430769, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:10,771 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:10,771 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39458 deadline: 1682391430769, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:10,771 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:10,771 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39536 deadline: 1682391430769, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:10,950 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/.tmp/C/65eebb4d60aa4ce0a416cb0b70f2c05f as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/C/65eebb4d60aa4ce0a416cb0b70f2c05f 2023-04-25 02:56:10,953 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in d6e38fead90b2a328d380bf5577ff0ce/C of d6e38fead90b2a328d380bf5577ff0ce into 65eebb4d60aa4ce0a416cb0b70f2c05f(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 02:56:10,954 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d6e38fead90b2a328d380bf5577ff0ce: 2023-04-25 02:56:10,954 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce., storeName=d6e38fead90b2a328d380bf5577ff0ce/C, priority=13, startTime=1682391370496; duration=0sec 2023-04-25 02:56:10,954 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 02:56:10,971 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:10,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39522 deadline: 1682391430969, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:10,972 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:10,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39526 deadline: 1682391430970, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:10,973 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:10,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39482 deadline: 1682391430971, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:10,974 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:10,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39458 deadline: 1682391430973, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:10,975 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:10,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39536 deadline: 1682391430973, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:11,025 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230425d7a9358fbeeb4633afd89389db1b82c8 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425d7a9358fbeeb4633afd89389db1b82c8 2023-04-25 02:56:11,274 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:11,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39522 deadline: 1682391431273, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:11,274 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:11,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39526 deadline: 1682391431273, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:11,278 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:11,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39482 deadline: 1682391431277, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:11,278 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:11,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39458 deadline: 1682391431277, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:11,278 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:11,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39536 deadline: 1682391431278, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:11,346 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/.tmp/A/e53a9f39bec94421ac6530cd94694dad as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/A/e53a9f39bec94421ac6530cd94694dad 2023-04-25 02:56:11,350 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in d6e38fead90b2a328d380bf5577ff0ce/A of d6e38fead90b2a328d380bf5577ff0ce into e53a9f39bec94421ac6530cd94694dad(size=25.7 K), total size for store is 25.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 02:56:11,350 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for d6e38fead90b2a328d380bf5577ff0ce: 2023-04-25 02:56:11,350 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce., storeName=d6e38fead90b2a328d380bf5577ff0ce/A, priority=13, startTime=1682391370496; duration=0sec 2023-04-25 02:56:11,350 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 02:56:11,430 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=203, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/.tmp/A/0c86fb4b21e34d7eaacef96eb63885eb 2023-04-25 02:56:11,440 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=203 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/.tmp/B/46bdb60b0b3e4fb090f24be4a7a6713c 2023-04-25 02:56:11,778 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:11,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39526 deadline: 1682391431777, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:11,780 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:11,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39522 deadline: 1682391431778, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:11,784 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:11,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39536 deadline: 1682391431781, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:11,784 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:11,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39482 deadline: 1682391431782, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:11,785 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:11,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39458 deadline: 1682391431783, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:11,849 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=203 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/.tmp/C/8a0db3ededdb4939a771361316e4b80b 2023-04-25 02:56:11,855 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/.tmp/A/0c86fb4b21e34d7eaacef96eb63885eb as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/A/0c86fb4b21e34d7eaacef96eb63885eb 2023-04-25 02:56:11,859 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/A/0c86fb4b21e34d7eaacef96eb63885eb, entries=150, sequenceid=203, filesize=25.3 K 2023-04-25 02:56:11,860 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/.tmp/B/46bdb60b0b3e4fb090f24be4a7a6713c as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/B/46bdb60b0b3e4fb090f24be4a7a6713c 2023-04-25 02:56:11,863 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/B/46bdb60b0b3e4fb090f24be4a7a6713c, entries=150, sequenceid=203, filesize=11.8 K 2023-04-25 02:56:11,864 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/.tmp/C/8a0db3ededdb4939a771361316e4b80b as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/C/8a0db3ededdb4939a771361316e4b80b 2023-04-25 02:56:11,867 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/C/8a0db3ededdb4939a771361316e4b80b, entries=150, sequenceid=203, filesize=11.8 K 2023-04-25 02:56:11,868 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for d6e38fead90b2a328d380bf5577ff0ce in 1266ms, sequenceid=203, compaction requested=false 2023-04-25 02:56:11,868 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d6e38fead90b2a328d380bf5577ff0ce: 2023-04-25 02:56:11,868 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. 2023-04-25 02:56:11,868 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:56:11,869 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:56:11,869 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:11,869 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:56:11,869 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:56:11,870 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:11,870 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:11,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:11,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:11,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:11,870 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:11,871 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:56:11,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:11,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:11,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:11,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:11,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:11,872 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:56:11,873 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3432519e[Count = 0] remaining members to acquire global barrier 2023-04-25 02:56:11,873 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:56:11,873 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:11,874 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:11,874 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:11,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:11,874 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:56:11,874 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:56:11,874 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:56:11,874 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:11,874 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:56:11,875 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:11,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:11,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:11,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:11,875 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:56:11,875 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:56:11,875 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:56:11,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:11,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:11,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:11,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:11,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:11,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:11,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:11,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:56:11,878 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:56:11,878 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:56:11,878 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:56:11,878 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:56:11,878 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:56:11,878 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:56:11,883 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:11,883 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:11,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:11,883 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:11,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:11,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:11,883 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:56:11,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:11,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:11,884 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:56:11,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:56:11,884 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:11,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:11,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:11,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:11,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:11,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:11,885 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:11,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:11,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:11,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:11,887 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:56:11,887 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:11,887 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:56:11,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:56:11,887 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:11,888 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:11,888 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:11,888 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:11,888 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:11,888 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:11,887 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:56:11,888 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:56:11,887 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:56:11,887 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:56:11,888 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:56:11,888 DEBUG [Thread-3656] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:56:11,888 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:56:11,888 DEBUG [Thread-3656] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:56:11,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:56:11,988 DEBUG [Thread-3656] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:56:11,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:56:11,990 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:56:11,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:56:11,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:56:11,996 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:56:11,996 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:56:11,996 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:56:11,996 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:56:11,996 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:11,996 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:11,997 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:56:11,997 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:11,997 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:56:11,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:56:11,998 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:11,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:11,998 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:56:11,998 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:11,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:56:11,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:11,999 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:56:11,999 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:56:11,999 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:56:11,999 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:56:11,999 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:56:11,999 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:56:11,999 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:56:11,999 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:56:11,999 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. 2023-04-25 02:56:11,999 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. started... 2023-04-25 02:56:11,999 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d6e38fead90b2a328d380bf5577ff0ce 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-25 02:56:12,000 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6e38fead90b2a328d380bf5577ff0ce, store=A 2023-04-25 02:56:12,000 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:56:12,000 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6e38fead90b2a328d380bf5577ff0ce, store=B 2023-04-25 02:56:12,000 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:56:12,000 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6e38fead90b2a328d380bf5577ff0ce, store=C 2023-04-25 02:56:12,000 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:56:12,014 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304253bcbd353bbf4400e95080076864fba33 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304253bcbd353bbf4400e95080076864fba33 2023-04-25 02:56:12,026 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=217, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/.tmp/A/7a771c486a5f45bf8de7bb7b803f88b0 2023-04-25 02:56:12,041 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=217 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/.tmp/B/82c89d03598e44eca2dee92de67c927b 2023-04-25 02:56:12,452 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=217 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/.tmp/C/adec822067b84a4c9fb3c85f376087e5 2023-04-25 02:56:12,457 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/.tmp/A/7a771c486a5f45bf8de7bb7b803f88b0 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/A/7a771c486a5f45bf8de7bb7b803f88b0 2023-04-25 02:56:12,461 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/A/7a771c486a5f45bf8de7bb7b803f88b0, entries=150, sequenceid=217, filesize=25.3 K 2023-04-25 02:56:12,462 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/.tmp/B/82c89d03598e44eca2dee92de67c927b as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/B/82c89d03598e44eca2dee92de67c927b 2023-04-25 02:56:12,465 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/B/82c89d03598e44eca2dee92de67c927b, entries=150, sequenceid=217, filesize=11.8 K 2023-04-25 02:56:12,466 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/.tmp/C/adec822067b84a4c9fb3c85f376087e5 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/C/adec822067b84a4c9fb3c85f376087e5 2023-04-25 02:56:12,469 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/C/adec822067b84a4c9fb3c85f376087e5, entries=150, sequenceid=217, filesize=11.8 K 2023-04-25 02:56:12,470 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=0 B/0 for d6e38fead90b2a328d380bf5577ff0ce in 471ms, sequenceid=217, compaction requested=true 2023-04-25 02:56:12,470 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d6e38fead90b2a328d380bf5577ff0ce: 2023-04-25 02:56:12,470 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. 2023-04-25 02:56:12,470 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:56:12,470 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:56:12,471 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:12,471 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:56:12,471 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:56:12,472 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:12,472 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:12,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:12,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:12,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:12,473 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:12,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:12,473 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:56:12,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:12,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:12,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:12,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:12,474 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:56:12,474 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5aab1330[Count = 0] remaining members to acquire global barrier 2023-04-25 02:56:12,474 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:56:12,474 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:12,476 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:12,477 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:12,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:12,477 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:56:12,477 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:56:12,477 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:12,477 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:56:12,477 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:56:12,478 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:56:12,478 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:56:12,478 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:56:12,478 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:12,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:12,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:12,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:12,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:12,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:12,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:12,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:12,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:12,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:12,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:12,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:56:12,481 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:56:12,481 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:56:12,481 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:56:12,481 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:56:12,481 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:56:12,481 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:56:12,483 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:12,483 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:12,483 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:56:12,483 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:12,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:12,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:12,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:12,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:12,483 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:56:12,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:56:12,483 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:12,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:12,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:12,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:12,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:12,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:12,485 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:12,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:12,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:12,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:12,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:12,487 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:12,487 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:12,487 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:56:12,487 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:12,487 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:56:12,487 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:56:12,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:56:12,487 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:56:12,487 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:56:12,487 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:12,487 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:56:12,487 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:12,487 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:12,487 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:56:12,487 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:12,487 DEBUG [Thread-3656] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:56:12,487 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:56:12,487 DEBUG [Thread-3656] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:56:12,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:56:12,588 DEBUG [Thread-3656] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:56:12,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:56:12,590 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:56:12,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:56:12,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:56:12,591 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:56:12,591 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:56:12,591 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:56:12,591 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:56:12,593 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:12,593 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:12,595 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:56:12,595 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:56:12,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:56:12,595 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:12,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:12,595 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:12,595 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:56:12,596 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:12,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:56:12,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:12,596 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:56:12,596 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:56:12,596 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:56:12,596 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:56:12,597 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:56:12,597 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:56:12,597 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:56:12,597 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:56:12,597 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. 2023-04-25 02:56:12,597 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. started... 2023-04-25 02:56:12,597 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d6e38fead90b2a328d380bf5577ff0ce: 2023-04-25 02:56:12,597 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. 2023-04-25 02:56:12,597 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:56:12,597 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:56:12,597 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:12,597 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:56:12,597 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:56:12,599 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:12,599 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:12,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:12,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:12,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:12,599 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:12,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:12,599 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:56:12,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:12,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:12,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:12,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:12,601 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:56:12,601 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1f97521b[Count = 0] remaining members to acquire global barrier 2023-04-25 02:56:12,601 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:56:12,601 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:12,602 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:12,602 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:12,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:12,602 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:56:12,602 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:56:12,602 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:56:12,602 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:12,602 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:56:12,603 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:12,603 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:56:12,603 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:56:12,603 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:56:12,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:12,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:12,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:12,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:12,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:12,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:12,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:12,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:12,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:12,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:12,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:56:12,606 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:56:12,606 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:56:12,606 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:56:12,606 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:56:12,606 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:56:12,606 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:56:12,609 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:12,609 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:12,609 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:12,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:12,609 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:56:12,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:12,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:12,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:12,609 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:56:12,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:56:12,609 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:12,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:12,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:12,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:12,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:12,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:12,610 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:12,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:12,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:12,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:12,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:12,613 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:56:12,613 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:56:12,613 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:56:12,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:56:12,613 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:12,613 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:12,613 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:12,613 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:56:12,613 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:12,613 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:12,613 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:56:12,613 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:56:12,613 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:56:12,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:56:12,613 DEBUG [Thread-3656] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:56:12,613 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:12,613 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:12,613 DEBUG [Thread-3656] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:56:12,614 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:56:12,714 DEBUG [Thread-3656] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:56:12,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:56:12,716 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:56:12,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:56:12,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:56:12,717 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:56:12,717 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:56:12,717 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:56:12,717 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:56:12,718 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:12,718 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:12,719 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:56:12,719 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:12,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:56:12,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:56:12,719 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:12,719 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:56:12,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:12,720 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:12,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:56:12,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:12,720 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:56:12,722 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:56:12,722 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:56:12,722 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:56:12,722 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:56:12,722 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:56:12,722 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:56:12,723 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:56:12,723 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. 2023-04-25 02:56:12,723 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. started... 2023-04-25 02:56:12,723 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d6e38fead90b2a328d380bf5577ff0ce: 2023-04-25 02:56:12,723 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. 2023-04-25 02:56:12,723 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:56:12,723 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:56:12,723 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:12,723 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:56:12,723 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:56:12,724 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:12,724 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:12,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:12,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:12,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:12,725 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:12,725 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:56:12,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:12,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:12,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:12,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:12,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:12,726 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:56:12,726 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1ec9b888[Count = 0] remaining members to acquire global barrier 2023-04-25 02:56:12,726 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:56:12,726 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:12,728 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:12,728 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:12,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:12,728 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:56:12,728 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:56:12,728 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:12,728 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:56:12,728 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:56:12,730 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:12,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:12,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:12,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:12,730 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:56:12,730 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:56:12,730 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:56:12,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:12,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:12,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:12,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:12,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:12,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:12,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:12,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:56:12,733 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:56:12,733 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:56:12,733 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:56:12,733 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:56:12,733 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:56:12,733 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:56:12,734 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:12,734 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:12,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:12,734 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:12,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:12,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:12,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:12,734 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:56:12,735 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:56:12,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:56:12,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:12,735 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:12,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:12,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:12,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:12,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:12,737 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:12,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:12,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:12,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:12,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:12,743 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:12,743 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:12,743 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:12,743 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:12,743 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:56:12,743 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:56:12,743 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:56:12,743 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:12,743 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:56:12,744 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:56:12,743 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:12,743 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:56:12,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:56:12,744 DEBUG [Thread-3656] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:56:12,743 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:56:12,744 DEBUG [Thread-3656] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:56:12,744 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:12,744 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:56:12,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:56:12,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(9158): Flush requested on d6e38fead90b2a328d380bf5577ff0ce 2023-04-25 02:56:12,816 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing d6e38fead90b2a328d380bf5577ff0ce 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-25 02:56:12,816 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6e38fead90b2a328d380bf5577ff0ce, store=A 2023-04-25 02:56:12,816 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:56:12,816 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6e38fead90b2a328d380bf5577ff0ce, store=B 2023-04-25 02:56:12,816 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:56:12,816 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6e38fead90b2a328d380bf5577ff0ce, store=C 2023-04-25 02:56:12,816 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:56:12,855 DEBUG [Thread-3656] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:56:12,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:56:12,857 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:56:12,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:56:12,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:56:12,858 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:56:12,859 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:56:12,859 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:56:12,859 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:56:12,859 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:12,859 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:12,860 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:56:12,860 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:56:12,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:56:12,861 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:12,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:12,861 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:12,861 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:56:12,861 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:12,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:56:12,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:12,862 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:56:12,862 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:56:12,862 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:56:12,862 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:56:12,862 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:56:12,862 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:56:12,862 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:56:12,862 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. 2023-04-25 02:56:12,862 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. started... 2023-04-25 02:56:12,862 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. as already flushing 2023-04-25 02:56:12,862 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. 2023-04-25 02:56:12,862 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:56:12,862 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:56:12,862 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:12,863 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:56:12,863 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:56:12,864 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:12,864 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:12,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:12,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:12,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:12,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:12,866 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682391372856, max timestamp=1682391372864Num uniques 859; 2023-04-25 02:56:12,866 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-25 02:56:12,866 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682391372856, max timestamp=1682391372864Num uniques 859; 2023-04-25 02:56:12,866 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-25 02:56:12,866 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682391372856, max timestamp=1682391372864Num uniques 859; 2023-04-25 02:56:12,866 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:12,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:12,866 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:56:12,866 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-25 02:56:12,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:12,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:12,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:12,868 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:56:12,868 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3851a46e[Count = 0] remaining members to acquire global barrier 2023-04-25 02:56:12,868 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:56:12,868 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:12,869 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:12,869 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:12,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:12,869 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:56:12,869 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:56:12,869 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:56:12,869 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:12,869 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:56:12,872 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:12,872 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:56:12,872 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:56:12,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:12,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:12,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:12,872 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:56:12,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:12,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:12,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:12,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:12,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:12,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:12,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:12,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:56:12,874 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:56:12,874 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:56:12,874 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:56:12,874 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:56:12,874 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:56:12,874 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:56:12,876 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:12,876 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:12,876 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:12,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:12,876 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:56:12,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:12,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:12,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:12,876 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:56:12,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:56:12,876 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:12,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:12,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:12,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:12,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:12,878 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:12,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:12,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:12,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:12,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:12,881 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:56:12,881 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:56:12,881 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:56:12,881 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:56:12,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:56:12,881 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:12,881 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:12,881 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:12,881 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:56:12,881 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:56:12,881 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:12,881 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:56:12,881 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:12,881 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:12,881 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:12,881 DEBUG [Thread-3656] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:56:12,881 DEBUG [Thread-3656] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:56:12,882 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:56:12,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:56:12,885 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:12,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39458 deadline: 1682391432881, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:12,887 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1682391372878, max timestamp=1682391372884Num uniques 1018; 2023-04-25 02:56:12,887 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1682391372878, max timestamp=1682391372884Num uniques 1018; 2023-04-25 02:56:12,887 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=12268, suffix heap size=133840,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=268 2023-04-25 02:56:12,887 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=12268, suffix heap size=133840,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=268 2023-04-25 02:56:12,888 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1682391372879, max timestamp=1682391372884Num uniques 1018; 2023-04-25 02:56:12,888 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-25 02:56:12,908 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:12,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39536 deadline: 1682391432904, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:12,908 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:12,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39526 deadline: 1682391432905, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:12,908 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:12,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39522 deadline: 1682391432905, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:12,910 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:12,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39482 deadline: 1682391432908, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:12,981 DEBUG [Thread-3656] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:56:12,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:56:12,983 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:56:12,984 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:56:12,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:56:12,985 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:56:12,985 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:56:12,985 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:56:12,985 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:56:12,985 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:12,985 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:12,986 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:56:12,986 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:12,986 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:56:12,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:56:12,987 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:12,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:12,987 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:56:12,987 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:12,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:56:12,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:12,988 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:56:12,988 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:56:12,988 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:56:12,988 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:56:12,988 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:56:12,988 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:56:12,988 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:56:12,988 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:56:12,988 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. 2023-04-25 02:56:12,988 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. started... 2023-04-25 02:56:12,989 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. as already flushing 2023-04-25 02:56:12,989 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. 2023-04-25 02:56:12,989 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:56:12,989 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:56:12,989 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:12,989 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:56:12,989 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:56:12,990 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:12,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39458 deadline: 1682391432988, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:12,991 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:12,991 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:12,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:12,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:12,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:12,991 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:12,991 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:56:12,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:12,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:12,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:12,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:12,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:12,992 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:56:12,992 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3ae102c7[Count = 0] remaining members to acquire global barrier 2023-04-25 02:56:12,992 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:56:12,992 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:12,993 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:12,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:12,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:12,994 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:56:12,994 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:56:12,994 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:56:12,994 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:12,994 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:56:12,995 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:56:12,995 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:12,995 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:56:12,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:12,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:12,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:12,995 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:56:12,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:12,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:12,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:12,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:12,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:12,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:12,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:12,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:56:12,997 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:56:12,997 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:56:12,998 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:56:12,998 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:56:12,998 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:56:12,998 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:56:12,999 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:12,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:12,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:12,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:12,999 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:12,999 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:56:12,999 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:12,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:12,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:13,000 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:56:13,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:56:13,000 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:13,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:13,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:13,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:13,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:13,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:13,001 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:13,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:13,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:13,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:13,004 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:13,004 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:13,004 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:13,004 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:13,004 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:56:13,004 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:56:13,005 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:56:13,004 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:56:13,004 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:13,004 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:56:13,004 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:56:13,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:56:13,005 DEBUG [Thread-3656] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:56:13,005 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:13,005 DEBUG [Thread-3656] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:56:13,005 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:13,006 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:56:13,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:56:13,013 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:13,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39536 deadline: 1682391433009, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:13,013 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:13,013 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:13,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39526 deadline: 1682391433009, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:13,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39522 deadline: 1682391433009, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:13,013 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:13,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39482 deadline: 1682391433011, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:13,105 DEBUG [Thread-3656] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:56:13,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:56:13,107 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:56:13,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:56:13,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:56:13,109 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:56:13,109 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:56:13,109 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:56:13,109 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:56:13,109 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:13,109 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:13,111 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:13,111 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:56:13,111 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:56:13,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:56:13,111 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:13,111 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:56:13,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:13,112 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:13,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:56:13,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:13,112 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:56:13,112 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:56:13,113 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:56:13,113 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:56:13,113 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:56:13,113 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:56:13,113 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:56:13,113 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:56:13,113 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. 2023-04-25 02:56:13,113 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. started... 2023-04-25 02:56:13,113 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. as already flushing 2023-04-25 02:56:13,113 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. 2023-04-25 02:56:13,113 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:56:13,113 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:56:13,113 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:13,113 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:56:13,113 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:56:13,115 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:13,115 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:13,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:13,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:13,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:13,116 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:13,116 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:56:13,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:13,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:13,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:13,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:13,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:13,117 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:56:13,117 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1f43980a[Count = 0] remaining members to acquire global barrier 2023-04-25 02:56:13,117 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:56:13,117 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:13,118 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:13,119 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:13,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:13,119 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:56:13,119 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:56:13,119 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:56:13,121 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:56:13,121 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:56:13,121 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:56:13,128 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:13,128 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:56:13,128 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:13,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:13,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:13,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:13,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:13,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:13,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:13,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:13,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:13,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:13,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:13,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:56:13,130 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:56:13,130 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:56:13,131 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:56:13,131 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:56:13,131 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:56:13,131 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:56:13,132 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:13,132 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:13,132 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:13,132 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:56:13,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:13,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:13,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:13,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:13,133 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:56:13,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:56:13,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:13,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:13,134 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:13,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:13,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:13,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:13,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:13,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:13,136 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:13,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:13,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:13,141 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:56:13,141 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:56:13,141 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:56:13,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:56:13,141 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:13,141 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:56:13,141 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:56:13,141 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:56:13,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:56:13,141 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:56:13,141 DEBUG [Thread-3656] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:56:13,142 DEBUG [Thread-3656] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:56:13,141 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:13,142 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:13,142 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:13,142 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:13,142 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:13,142 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:13,192 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:13,193 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39458 deadline: 1682391433191, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:13,216 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:13,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39536 deadline: 1682391433214, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:13,217 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:13,217 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39522 deadline: 1682391433214, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:13,217 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:13,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39526 deadline: 1682391433215, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:13,218 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:13,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39482 deadline: 1682391433215, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:13,229 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230425d1015843fc9a4d2a8a857d0cab6b2085 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425d1015843fc9a4d2a8a857d0cab6b2085 2023-04-25 02:56:13,242 DEBUG [Thread-3656] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:56:13,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:56:13,244 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:56:13,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:56:13,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:56:13,248 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:56:13,248 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:56:13,248 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:56:13,248 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:56:13,249 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:13,249 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:13,250 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=229, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/.tmp/A/591d0a4903bf46e9b3bcacc9b62bf87e 2023-04-25 02:56:13,257 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:56:13,257 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:13,257 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:56:13,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:56:13,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:13,258 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:13,258 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:13,258 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:56:13,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:56:13,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:13,258 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:56:13,259 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:56:13,259 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:56:13,259 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:56:13,259 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:56:13,259 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:56:13,262 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:56:13,262 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:56:13,262 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. 2023-04-25 02:56:13,262 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. started... 2023-04-25 02:56:13,262 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. as already flushing 2023-04-25 02:56:13,262 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. 2023-04-25 02:56:13,262 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:56:13,262 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:56:13,262 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:13,262 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:56:13,262 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:56:13,264 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:13,264 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:13,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:13,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:13,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:13,264 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:13,264 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:56:13,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:13,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:13,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:13,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:13,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:13,266 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:56:13,266 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4c1d6e92[Count = 0] remaining members to acquire global barrier 2023-04-25 02:56:13,266 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:56:13,266 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:13,267 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:13,267 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:13,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:13,267 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:56:13,267 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:56:13,267 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:56:13,267 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:13,267 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:56:13,268 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:13,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:13,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:13,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:13,268 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:56:13,268 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:56:13,268 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:56:13,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:13,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:13,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:13,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:13,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:13,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:13,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:13,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:56:13,278 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:56:13,278 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:56:13,278 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:56:13,278 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:56:13,278 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:56:13,278 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:56:13,280 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:13,280 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:56:13,280 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:13,280 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:13,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:13,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:13,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:13,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:13,280 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:56:13,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:56:13,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:13,281 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:13,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:13,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:13,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:13,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:13,282 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:13,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:13,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:13,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:13,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:13,285 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:56:13,285 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:56:13,285 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:56:13,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:56:13,285 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:56:13,285 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:56:13,285 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:56:13,285 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:56:13,285 DEBUG [Thread-3656] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:56:13,285 DEBUG [Thread-3656] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:56:13,285 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:13,285 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:56:13,285 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:13,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:56:13,285 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:13,285 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:13,285 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:13,285 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:13,285 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:13,385 DEBUG [Thread-3656] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:56:13,386 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:56:13,387 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:56:13,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:56:13,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:56:13,389 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:56:13,389 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:56:13,389 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:56:13,389 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:56:13,389 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:13,389 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:13,390 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:56:13,390 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:56:13,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:56:13,390 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:13,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:13,391 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:13,391 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:56:13,391 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:13,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:56:13,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:13,391 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:56:13,392 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:56:13,392 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:56:13,392 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:56:13,392 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:56:13,392 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:56:13,392 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:56:13,392 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:56:13,392 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. 2023-04-25 02:56:13,392 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. started... 2023-04-25 02:56:13,392 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. as already flushing 2023-04-25 02:56:13,392 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. 2023-04-25 02:56:13,392 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:56:13,392 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:56:13,392 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:13,392 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:56:13,392 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:56:13,394 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:13,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:13,394 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:13,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:13,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:13,394 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:13,394 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:56:13,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:13,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:13,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:13,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:13,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:13,395 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:56:13,395 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@79dac1ab[Count = 0] remaining members to acquire global barrier 2023-04-25 02:56:13,395 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:56:13,395 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:13,397 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:13,397 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:13,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:13,398 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:56:13,398 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:56:13,398 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:56:13,398 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:13,398 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:56:13,399 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:13,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:13,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:13,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:13,399 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:56:13,399 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:56:13,399 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:56:13,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:13,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:13,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:13,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:13,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:13,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:13,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:13,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:56:13,401 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:56:13,401 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:56:13,401 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:56:13,401 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:56:13,401 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:56:13,401 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:56:13,403 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:13,403 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:13,403 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:13,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:13,403 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:56:13,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:13,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:13,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:13,403 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:56:13,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:56:13,404 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:13,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:13,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:13,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:13,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:13,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:13,405 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:13,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:13,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:13,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:13,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:13,407 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:56:13,407 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:56:13,407 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:56:13,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:56:13,407 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:13,407 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:13,407 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:13,407 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:56:13,407 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:56:13,407 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:56:13,407 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:13,407 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:13,407 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:13,407 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:13,407 DEBUG [Thread-3656] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:56:13,407 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:56:13,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:56:13,408 DEBUG [Thread-3656] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:56:13,408 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:56:13,496 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:13,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39458 deadline: 1682391433496, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:13,508 DEBUG [Thread-3656] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:56:13,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:56:13,510 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:56:13,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:56:13,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:56:13,511 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:56:13,511 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:56:13,511 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:56:13,511 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:56:13,512 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:13,512 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:13,514 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:56:13,514 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:13,514 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:56:13,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:56:13,515 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:13,515 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:56:13,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:13,515 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:13,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:56:13,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:13,516 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:56:13,516 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:56:13,516 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:56:13,516 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:56:13,516 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:56:13,516 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:56:13,516 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:56:13,516 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:56:13,516 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. 2023-04-25 02:56:13,516 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. started... 2023-04-25 02:56:13,516 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. as already flushing 2023-04-25 02:56:13,516 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. 2023-04-25 02:56:13,517 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:56:13,517 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:56:13,517 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:13,517 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:56:13,517 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:56:13,519 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:13,519 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:13,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39522 deadline: 1682391433518, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:13,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:13,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:13,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:13,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:13,520 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:13,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:13,520 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:13,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39482 deadline: 1682391433520, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:13,520 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:13,520 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:56:13,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:13,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:13,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:13,521 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:13,521 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:56:13,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39536 deadline: 1682391433520, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:13,522 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4b9f136d[Count = 0] remaining members to acquire global barrier 2023-04-25 02:56:13,522 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:13,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39526 deadline: 1682391433521, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:13,522 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:56:13,522 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:13,523 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:13,523 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:13,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:13,523 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:56:13,523 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:56:13,523 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:56:13,523 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:13,523 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:56:13,524 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:13,524 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:56:13,525 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:56:13,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:13,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:13,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:13,525 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:56:13,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:13,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:13,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:13,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:13,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:13,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:13,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:13,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:56:13,527 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:56:13,527 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:56:13,527 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:56:13,527 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:56:13,527 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:56:13,527 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:56:13,529 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:13,529 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:56:13,529 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:13,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:13,529 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:13,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:13,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:13,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:13,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:13,530 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:56:13,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:56:13,530 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:13,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:13,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:13,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:13,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:13,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:13,531 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:13,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:13,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:13,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:13,534 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:13,534 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:13,534 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:13,534 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:13,534 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:13,534 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:56:13,534 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:56:13,534 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:56:13,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:56:13,534 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:56:13,535 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:56:13,534 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:13,535 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:13,534 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:56:13,535 DEBUG [Thread-3656] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:56:13,535 DEBUG [Thread-3656] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:56:13,535 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:56:13,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:56:13,635 DEBUG [Thread-3656] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:56:13,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:56:13,639 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:56:13,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:56:13,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:56:13,643 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:56:13,643 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:56:13,643 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:56:13,643 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:56:13,644 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:13,644 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:13,645 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:56:13,645 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:13,645 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:56:13,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:56:13,646 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:13,646 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:56:13,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:13,646 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:13,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:56:13,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:13,647 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:56:13,647 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:56:13,647 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:56:13,647 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:56:13,647 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:56:13,647 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:56:13,647 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:56:13,647 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:56:13,647 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. 2023-04-25 02:56:13,647 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. started... 2023-04-25 02:56:13,647 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. as already flushing 2023-04-25 02:56:13,648 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. 2023-04-25 02:56:13,648 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:56:13,648 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:56:13,648 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:13,648 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:56:13,648 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:56:13,652 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:13,652 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:13,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:13,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:13,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:13,652 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:13,652 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:56:13,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:13,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:13,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:13,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:13,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:13,654 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:56:13,654 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@481f51dc[Count = 0] remaining members to acquire global barrier 2023-04-25 02:56:13,654 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:56:13,654 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:13,655 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:13,655 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:13,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:13,655 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:56:13,655 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:56:13,655 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:56:13,656 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:13,656 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:56:13,657 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:13,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:13,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:13,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:13,657 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:56:13,657 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:56:13,657 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:56:13,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:13,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:13,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:13,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:13,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:13,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:13,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:13,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:56:13,659 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:56:13,659 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:56:13,660 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:56:13,660 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:56:13,660 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:56:13,660 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:56:13,661 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:13,661 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:13,661 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:13,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:13,661 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:56:13,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:13,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:13,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:13,662 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:56:13,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:56:13,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:13,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:13,662 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:13,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:13,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:13,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:13,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:13,663 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:13,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:13,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:13,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:13,666 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:13,666 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:13,666 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:13,666 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:13,666 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:13,666 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:56:13,666 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:13,666 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:56:13,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:56:13,667 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:56:13,667 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:56:13,667 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:56:13,666 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:56:13,666 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:56:13,666 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:13,667 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:56:13,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:56:13,667 DEBUG [Thread-3656] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:56:13,667 DEBUG [Thread-3656] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:56:13,673 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=229 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/.tmp/B/18bee31ffbca4c088a357b9d7c613e53 2023-04-25 02:56:13,767 DEBUG [Thread-3656] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:56:13,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:56:13,769 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:56:13,771 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:56:13,771 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:56:13,771 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:56:13,771 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:56:13,771 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:56:13,771 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:56:13,772 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:13,772 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:13,773 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:56:13,773 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:56:13,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:56:13,773 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:13,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:13,774 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:13,774 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:13,774 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:56:13,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:56:13,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:13,774 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:56:13,774 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:56:13,774 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:56:13,774 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:56:13,774 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:56:13,775 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:56:13,775 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:56:13,775 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:56:13,775 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. 2023-04-25 02:56:13,775 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. started... 2023-04-25 02:56:13,775 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. as already flushing 2023-04-25 02:56:13,775 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. 2023-04-25 02:56:13,775 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:56:13,775 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:56:13,775 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:13,775 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:56:13,775 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:56:13,777 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:13,777 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:13,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:13,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:13,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:13,777 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:13,777 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:56:13,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:13,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:13,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:13,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:13,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:13,778 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:56:13,778 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6fd1cfb3[Count = 0] remaining members to acquire global barrier 2023-04-25 02:56:13,778 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:56:13,778 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:13,780 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:13,780 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:13,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:13,780 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:56:13,780 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:56:13,780 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:56:13,780 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:13,780 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:56:13,782 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:13,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:13,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:13,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:13,783 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:56:13,783 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:56:13,783 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:56:13,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:13,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:13,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:13,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:13,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:13,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:13,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:13,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:56:13,786 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:56:13,786 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:56:13,786 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:56:13,786 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:56:13,786 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:56:13,786 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:56:13,788 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:13,788 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:13,788 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:56:13,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:13,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:13,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:13,788 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:13,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:13,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:13,789 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:56:13,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:56:13,789 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:13,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:13,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:13,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:13,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:13,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:13,790 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:13,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:13,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:13,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:13,793 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:13,793 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:13,793 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:56:13,793 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:56:13,793 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:13,793 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:56:13,793 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:56:13,793 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:13,793 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:56:13,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:56:13,793 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:56:13,793 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:13,793 DEBUG [Thread-3656] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:56:13,793 DEBUG [Thread-3656] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:56:13,793 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:13,793 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:13,793 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:56:13,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:56:13,893 DEBUG [Thread-3656] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:56:13,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:56:13,895 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:56:13,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:56:13,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:56:13,897 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:56:13,897 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:56:13,897 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:56:13,897 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:56:13,897 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:13,897 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:13,898 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:56:13,898 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:13,898 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:56:13,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:56:13,899 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:13,899 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:56:13,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:13,899 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:13,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:56:13,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:13,899 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:56:13,900 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:56:13,900 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:56:13,900 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:56:13,900 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:56:13,900 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:56:13,900 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:56:13,900 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:56:13,900 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. 2023-04-25 02:56:13,900 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. started... 2023-04-25 02:56:13,900 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. as already flushing 2023-04-25 02:56:13,900 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. 2023-04-25 02:56:13,900 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:56:13,900 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:56:13,900 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:13,900 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:56:13,900 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:56:13,903 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:13,903 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:13,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:13,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:13,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:13,903 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:13,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:13,903 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:56:13,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:13,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:13,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:13,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:13,904 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:56:13,904 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@40299c7b[Count = 0] remaining members to acquire global barrier 2023-04-25 02:56:13,904 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:56:13,904 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:13,905 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:13,905 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:13,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:13,906 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:56:13,906 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:56:13,906 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:56:13,906 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:13,906 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:56:13,907 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:13,907 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:56:13,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:13,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:13,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:13,907 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:56:13,907 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:56:13,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:13,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:13,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:13,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:13,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:13,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:13,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:13,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:56:13,912 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:56:13,912 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:56:13,912 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:56:13,912 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:56:13,912 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:56:13,912 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:56:13,914 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:13,914 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:13,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:13,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:13,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:13,914 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:56:13,914 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:13,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:13,914 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:56:13,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:13,914 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:13,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:56:13,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:13,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:13,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:13,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:13,916 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:13,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:13,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:13,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:13,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:13,918 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:13,918 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:13,918 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:13,918 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:56:13,918 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:56:13,918 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:13,918 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:13,918 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:56:13,919 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:56:13,918 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:56:13,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:56:13,919 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:13,919 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:56:13,919 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:56:13,919 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:13,919 DEBUG [Thread-3656] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:56:13,919 DEBUG [Thread-3656] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:56:13,919 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:56:13,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:56:14,003 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:14,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39458 deadline: 1682391434001, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:14,019 DEBUG [Thread-3656] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:56:14,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:56:14,021 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:56:14,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:56:14,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:56:14,022 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:56:14,022 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:56:14,023 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:56:14,023 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:56:14,023 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:14,023 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:14,024 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:56:14,024 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:56:14,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:56:14,024 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:14,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:14,025 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:14,025 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:56:14,025 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:14,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:56:14,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:14,025 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:56:14,026 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:14,026 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:56:14,026 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:56:14,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39526 deadline: 1682391434023, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:14,026 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:56:14,026 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:56:14,026 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:56:14,026 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:14,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39522 deadline: 1682391434024, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:14,026 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:56:14,026 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:56:14,026 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. 2023-04-25 02:56:14,026 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. started... 2023-04-25 02:56:14,027 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. as already flushing 2023-04-25 02:56:14,027 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. 2023-04-25 02:56:14,027 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:56:14,027 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:56:14,027 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:14,027 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:56:14,027 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:56:14,027 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:14,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39482 deadline: 1682391434025, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:14,028 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:14,028 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39536 deadline: 1682391434027, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:14,028 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:14,028 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:14,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:14,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:14,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:14,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:14,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:14,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:14,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:14,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:14,030 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:56:14,030 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@739efb68[Count = 0] remaining members to acquire global barrier 2023-04-25 02:56:14,030 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:56:14,030 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:14,034 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(162): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:14,034 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:14,034 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:56:14,034 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:56:14,035 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:14,035 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:56:14,035 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:56:14,035 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:56:14,036 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:14,036 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:56:14,036 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:56:14,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:14,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:14,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:14,036 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:56:14,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:14,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:14,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:14,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:14,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:14,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:14,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:14,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:56:14,038 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:56:14,038 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:56:14,038 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:56:14,039 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:56:14,039 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:56:14,039 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:56:14,040 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:14,040 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:14,040 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:56:14,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:14,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:14,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:14,040 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:14,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:14,040 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:56:14,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:56:14,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:14,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:14,041 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:14,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:14,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:14,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:14,043 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:14,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:14,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:14,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:14,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:14,045 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:14,045 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:14,045 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:14,045 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:14,045 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:14,045 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:14,045 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:14,045 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:56:14,045 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:14,045 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:56:14,045 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:56:14,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:56:14,045 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:56:14,045 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:56:14,045 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:56:14,046 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:56:14,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:56:14,045 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:56:14,046 DEBUG [Thread-3656] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:56:14,046 DEBUG [Thread-3656] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:56:14,084 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=229 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/.tmp/C/36b20ab738624ef28598ba7e9b4909cb 2023-04-25 02:56:14,088 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/.tmp/A/591d0a4903bf46e9b3bcacc9b62bf87e as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/A/591d0a4903bf46e9b3bcacc9b62bf87e 2023-04-25 02:56:14,091 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/A/591d0a4903bf46e9b3bcacc9b62bf87e, entries=200, sequenceid=229, filesize=32.1 K 2023-04-25 02:56:14,092 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/.tmp/B/18bee31ffbca4c088a357b9d7c613e53 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/B/18bee31ffbca4c088a357b9d7c613e53 2023-04-25 02:56:14,095 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/B/18bee31ffbca4c088a357b9d7c613e53, entries=150, sequenceid=229, filesize=11.8 K 2023-04-25 02:56:14,095 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/.tmp/C/36b20ab738624ef28598ba7e9b4909cb as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/C/36b20ab738624ef28598ba7e9b4909cb 2023-04-25 02:56:14,099 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/C/36b20ab738624ef28598ba7e9b4909cb, entries=150, sequenceid=229, filesize=11.8 K 2023-04-25 02:56:14,100 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=138.65 KB/141980 for d6e38fead90b2a328d380bf5577ff0ce in 1284ms, sequenceid=229, compaction requested=true 2023-04-25 02:56:14,100 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for d6e38fead90b2a328d380bf5577ff0ce: 2023-04-25 02:56:14,100 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 02:56:14,100 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-25 02:56:14,100 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-25 02:56:14,100 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 02:56:14,100 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 02:56:14,101 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 110950 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-25 02:56:14,101 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HStore(1912): d6e38fead90b2a328d380bf5577ff0ce/A is initiating minor compaction (all files) 2023-04-25 02:56:14,101 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of d6e38fead90b2a328d380bf5577ff0ce/A in TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. 2023-04-25 02:56:14,101 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/A/e53a9f39bec94421ac6530cd94694dad, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/A/0c86fb4b21e34d7eaacef96eb63885eb, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/A/7a771c486a5f45bf8de7bb7b803f88b0, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/A/591d0a4903bf46e9b3bcacc9b62bf87e] into tmpdir=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/.tmp, totalSize=108.3 K 2023-04-25 02:56:14,102 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48520 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-25 02:56:14,102 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HStore(1912): d6e38fead90b2a328d380bf5577ff0ce/B is initiating minor compaction (all files) 2023-04-25 02:56:14,102 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d6e38fead90b2a328d380bf5577ff0ce/B in TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. 2023-04-25 02:56:14,102 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/B/7cec68bee0994e788f89f2e393d4fd6d, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/B/46bdb60b0b3e4fb090f24be4a7a6713c, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/B/82c89d03598e44eca2dee92de67c927b, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/B/18bee31ffbca4c088a357b9d7c613e53] into tmpdir=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/.tmp, totalSize=47.4 K 2023-04-25 02:56:14,102 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting e53a9f39bec94421ac6530cd94694dad, keycount=150, bloomtype=ROW, size=25.7 K, encoding=NONE, compression=NONE, seqNum=182, earliestPutTs=1682391369719 2023-04-25 02:56:14,103 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting 7cec68bee0994e788f89f2e393d4fd6d, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=182, earliestPutTs=1682391369719 2023-04-25 02:56:14,103 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting 0c86fb4b21e34d7eaacef96eb63885eb, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=203, earliestPutTs=1682391370045 2023-04-25 02:56:14,103 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting 46bdb60b0b3e4fb090f24be4a7a6713c, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=203, earliestPutTs=1682391370045 2023-04-25 02:56:14,103 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting 7a771c486a5f45bf8de7bb7b803f88b0, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=217, earliestPutTs=1682391370660 2023-04-25 02:56:14,103 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting 82c89d03598e44eca2dee92de67c927b, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=217, earliestPutTs=1682391370660 2023-04-25 02:56:14,103 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting 591d0a4903bf46e9b3bcacc9b62bf87e, keycount=200, bloomtype=ROW, size=32.1 K, encoding=NONE, compression=NONE, seqNum=229, earliestPutTs=1682391372799 2023-04-25 02:56:14,104 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting 18bee31ffbca4c088a357b9d7c613e53, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=229, earliestPutTs=1682391372799 2023-04-25 02:56:14,113 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] throttle.PressureAwareThroughputController(145): d6e38fead90b2a328d380bf5577ff0ce#B#compaction#529 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 53.85 MB/second 2023-04-25 02:56:14,117 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] throttle.PressureAwareThroughputController(145): d6e38fead90b2a328d380bf5577ff0ce#A#compaction#528 average throughput is 3.29 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-25 02:56:14,138 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/.tmp/A/435557a72796411297823d2e5692300e as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/A/435557a72796411297823d2e5692300e 2023-04-25 02:56:14,142 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in d6e38fead90b2a328d380bf5577ff0ce/A of d6e38fead90b2a328d380bf5577ff0ce into 435557a72796411297823d2e5692300e(size=25.8 K), total size for store is 25.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 02:56:14,142 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for d6e38fead90b2a328d380bf5577ff0ce: 2023-04-25 02:56:14,142 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce., storeName=d6e38fead90b2a328d380bf5577ff0ce/A, priority=12, startTime=1682391374100; duration=0sec 2023-04-25 02:56:14,142 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 02:56:14,142 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-25 02:56:14,144 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48520 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-25 02:56:14,144 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HStore(1912): d6e38fead90b2a328d380bf5577ff0ce/C is initiating minor compaction (all files) 2023-04-25 02:56:14,144 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of d6e38fead90b2a328d380bf5577ff0ce/C in TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. 2023-04-25 02:56:14,144 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/C/65eebb4d60aa4ce0a416cb0b70f2c05f, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/C/8a0db3ededdb4939a771361316e4b80b, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/C/adec822067b84a4c9fb3c85f376087e5, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/C/36b20ab738624ef28598ba7e9b4909cb] into tmpdir=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/.tmp, totalSize=47.4 K 2023-04-25 02:56:14,144 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting 65eebb4d60aa4ce0a416cb0b70f2c05f, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=182, earliestPutTs=1682391369719 2023-04-25 02:56:14,145 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting 8a0db3ededdb4939a771361316e4b80b, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=203, earliestPutTs=1682391370045 2023-04-25 02:56:14,145 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting adec822067b84a4c9fb3c85f376087e5, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=217, earliestPutTs=1682391370660 2023-04-25 02:56:14,145 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting 36b20ab738624ef28598ba7e9b4909cb, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=229, earliestPutTs=1682391372799 2023-04-25 02:56:14,146 DEBUG [Thread-3656] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:56:14,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:56:14,148 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:56:14,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:56:14,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:56:14,150 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:56:14,150 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:56:14,150 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:56:14,150 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:56:14,150 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:14,150 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:14,152 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] throttle.PressureAwareThroughputController(145): d6e38fead90b2a328d380bf5577ff0ce#C#compaction#530 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-25 02:56:14,152 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:14,153 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:56:14,153 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:14,153 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:56:14,153 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:56:14,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:56:14,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:14,154 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:14,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:56:14,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:14,154 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:56:14,155 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:56:14,155 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:56:14,155 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:56:14,155 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:56:14,155 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:56:14,156 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:56:14,156 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:56:14,157 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. 2023-04-25 02:56:14,157 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. started... 2023-04-25 02:56:14,157 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d6e38fead90b2a328d380bf5577ff0ce 3/3 column families, dataSize=138.65 KB heapSize=389.61 KB 2023-04-25 02:56:14,157 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6e38fead90b2a328d380bf5577ff0ce, store=A 2023-04-25 02:56:14,157 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 02:56:14,157 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6e38fead90b2a328d380bf5577ff0ce, store=B 2023-04-25 02:56:14,157 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 02:56:14,157 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6e38fead90b2a328d380bf5577ff0ce, store=C 2023-04-25 02:56:14,157 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 02:56:14,160 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/.tmp/C/0fcbff231ef140a79b687e50fc9b047d as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/C/0fcbff231ef140a79b687e50fc9b047d 2023-04-25 02:56:14,168 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in d6e38fead90b2a328d380bf5577ff0ce/C of d6e38fead90b2a328d380bf5577ff0ce into 0fcbff231ef140a79b687e50fc9b047d(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 02:56:14,168 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for d6e38fead90b2a328d380bf5577ff0ce: 2023-04-25 02:56:14,168 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce., storeName=d6e38fead90b2a328d380bf5577ff0ce/C, priority=12, startTime=1682391374100; duration=0sec 2023-04-25 02:56:14,168 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 02:56:14,178 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230425f4558294ca2b41f48b498694ecc7cba2 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425f4558294ca2b41f48b498694ecc7cba2 2023-04-25 02:56:14,190 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=283, memsize=47.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/.tmp/A/1d6084fa00f646f4b04857d7ae1bcf77 2023-04-25 02:56:14,526 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/.tmp/B/9362dadfefb742f7a473b4f8e3e5c47a as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/B/9362dadfefb742f7a473b4f8e3e5c47a 2023-04-25 02:56:14,530 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in d6e38fead90b2a328d380bf5577ff0ce/B of d6e38fead90b2a328d380bf5577ff0ce into 9362dadfefb742f7a473b4f8e3e5c47a(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 02:56:14,530 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d6e38fead90b2a328d380bf5577ff0ce: 2023-04-25 02:56:14,530 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce., storeName=d6e38fead90b2a328d380bf5577ff0ce/B, priority=12, startTime=1682391374100; duration=0sec 2023-04-25 02:56:14,530 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 02:56:14,606 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=283 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/.tmp/B/3ce395b9822f4c0da488ea66c3782d9c 2023-04-25 02:56:14,620 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=283 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/.tmp/C/e275a85e2bf44fd19839f67df8bfb4d8 2023-04-25 02:56:14,624 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/.tmp/A/1d6084fa00f646f4b04857d7ae1bcf77 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/A/1d6084fa00f646f4b04857d7ae1bcf77 2023-04-25 02:56:14,628 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/A/1d6084fa00f646f4b04857d7ae1bcf77, entries=150, sequenceid=283, filesize=25.4 K 2023-04-25 02:56:14,629 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/.tmp/B/3ce395b9822f4c0da488ea66c3782d9c as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/B/3ce395b9822f4c0da488ea66c3782d9c 2023-04-25 02:56:14,632 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/B/3ce395b9822f4c0da488ea66c3782d9c, entries=150, sequenceid=283, filesize=11.9 K 2023-04-25 02:56:14,633 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/.tmp/C/e275a85e2bf44fd19839f67df8bfb4d8 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/C/e275a85e2bf44fd19839f67df8bfb4d8 2023-04-25 02:56:14,636 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/C/e275a85e2bf44fd19839f67df8bfb4d8, entries=150, sequenceid=283, filesize=11.9 K 2023-04-25 02:56:14,637 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~138.65 KB/141980, heapSize ~389.56 KB/398912, currentSize=0 B/0 for d6e38fead90b2a328d380bf5577ff0ce in 480ms, sequenceid=283, compaction requested=false 2023-04-25 02:56:14,637 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d6e38fead90b2a328d380bf5577ff0ce: 2023-04-25 02:56:14,637 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. 2023-04-25 02:56:14,637 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:56:14,637 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:56:14,637 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:14,637 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:56:14,637 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:56:14,639 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:14,639 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:14,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:14,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:14,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:14,639 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:14,639 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:56:14,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:14,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:14,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:14,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:14,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:14,641 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:56:14,641 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@41bdd576[Count = 0] remaining members to acquire global barrier 2023-04-25 02:56:14,641 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:56:14,641 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:14,642 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:14,642 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:14,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:14,642 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:56:14,642 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:56:14,642 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:56:14,642 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:14,642 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:56:14,643 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:56:14,643 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:14,643 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:56:14,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:14,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:14,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:14,644 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:56:14,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:14,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:14,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:14,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:14,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:14,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:14,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:14,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:56:14,646 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:56:14,646 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:56:14,647 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:56:14,647 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:56:14,647 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:56:14,647 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:56:14,649 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:14,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:14,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:14,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:14,650 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:14,650 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:56:14,650 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:14,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:14,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:14,650 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:14,650 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:56:14,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:56:14,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:14,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:14,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:14,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:14,651 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:14,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:14,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:14,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:14,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:14,654 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:56:14,654 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:14,654 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:56:14,654 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:56:14,654 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:56:14,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:56:14,654 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:56:14,654 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:56:14,654 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:56:14,654 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:14,654 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:14,654 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:56:14,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:56:14,654 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:14,654 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:14,654 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:14,654 DEBUG [Thread-3656] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:56:14,654 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:14,654 DEBUG [Thread-3656] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:56:14,755 DEBUG [Thread-3656] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:56:14,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:56:14,757 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:56:14,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:56:14,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:56:14,759 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:56:14,759 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:56:14,759 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:56:14,759 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:56:14,759 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:14,759 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:14,760 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:14,760 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:56:14,761 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:56:14,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:56:14,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:14,761 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:14,761 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:56:14,761 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:14,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:56:14,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:14,762 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:56:14,762 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:56:14,762 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:56:14,762 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:56:14,762 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:56:14,762 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:56:14,762 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:56:14,762 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:56:14,762 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. 2023-04-25 02:56:14,762 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. started... 2023-04-25 02:56:14,762 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d6e38fead90b2a328d380bf5577ff0ce: 2023-04-25 02:56:14,763 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. 2023-04-25 02:56:14,763 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:56:14,763 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:56:14,763 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:14,763 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:56:14,763 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:56:14,764 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:14,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:14,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:14,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:14,764 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:14,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:14,765 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:14,765 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:56:14,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:14,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:14,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:14,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:14,766 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:56:14,766 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@207036a5[Count = 0] remaining members to acquire global barrier 2023-04-25 02:56:14,766 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:56:14,766 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:14,768 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:14,768 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:14,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:14,768 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:14,768 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:56:14,768 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:56:14,768 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:56:14,768 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:56:14,770 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:14,770 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:56:14,770 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:56:14,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:14,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:14,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:14,770 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:56:14,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:14,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:14,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:14,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:14,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:14,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:14,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:14,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:56:14,772 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:56:14,772 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:56:14,772 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:56:14,772 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:56:14,772 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:56:14,773 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:56:14,774 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:14,774 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:14,774 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:14,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:14,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:14,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:14,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:14,774 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:56:14,774 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:56:14,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:56:14,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:14,775 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:14,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:14,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:14,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:14,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:14,776 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:14,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:14,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:14,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:14,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:14,778 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:14,778 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:14,778 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:14,778 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:14,778 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:56:14,778 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:56:14,778 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:56:14,778 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:56:14,778 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:14,778 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:56:14,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:56:14,778 DEBUG [Thread-3656] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:56:14,778 DEBUG [Thread-3656] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:56:14,778 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:14,779 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:56:14,778 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:56:14,779 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:56:14,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:56:14,779 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:14,879 DEBUG [Thread-3656] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:56:14,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:56:14,881 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:56:14,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:56:14,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:56:14,882 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:56:14,882 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:56:14,882 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:56:14,882 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:56:14,883 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:14,883 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:14,885 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:56:14,885 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:14,885 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:56:14,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:56:14,886 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:14,886 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:56:14,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:14,886 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:14,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:56:14,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:14,886 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:56:14,887 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:56:14,887 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:56:14,887 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:56:14,887 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:56:14,887 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:56:14,887 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:56:14,887 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:56:14,887 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. 2023-04-25 02:56:14,887 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. started... 2023-04-25 02:56:14,887 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d6e38fead90b2a328d380bf5577ff0ce: 2023-04-25 02:56:14,887 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. 2023-04-25 02:56:14,887 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:56:14,887 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:56:14,887 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:14,888 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:56:14,888 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:56:14,889 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:14,889 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:14,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:14,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:14,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:14,889 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:14,889 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:56:14,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:14,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:14,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:14,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:14,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:14,891 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:56:14,891 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2c36ac94[Count = 0] remaining members to acquire global barrier 2023-04-25 02:56:14,891 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:56:14,891 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:14,892 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:14,892 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:14,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:14,892 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:56:14,892 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:56:14,892 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:56:14,892 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:14,892 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:56:14,898 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:14,898 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:56:14,898 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:56:14,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:14,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:14,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:14,898 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:56:14,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:14,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:14,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:14,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:14,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:14,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:14,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:14,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:56:14,901 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:56:14,901 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:56:14,901 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:56:14,901 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:56:14,901 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:56:14,901 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:56:14,903 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:14,903 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:14,903 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:14,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:14,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:14,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:14,903 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:56:14,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:14,903 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:14,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:14,903 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:56:14,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:56:14,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:14,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:14,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:14,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:14,905 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:14,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:14,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:14,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:14,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:14,907 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:56:14,907 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:56:14,907 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:14,907 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:56:14,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:56:14,907 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:56:14,907 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:14,907 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:56:14,908 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:56:14,907 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:56:14,908 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:56:14,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:56:14,908 DEBUG [Thread-3656] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:56:14,908 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:14,908 DEBUG [Thread-3656] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:56:14,908 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:14,908 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:14,908 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:14,908 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:15,008 DEBUG [Thread-3656] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:56:15,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:56:15,010 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:56:15,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:56:15,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:56:15,012 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:56:15,012 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:56:15,012 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:56:15,012 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:56:15,012 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:15,012 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:15,013 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:56:15,013 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:15,014 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:56:15,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:56:15,014 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:15,014 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:56:15,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:15,014 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:15,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:56:15,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:15,015 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:56:15,015 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:56:15,015 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:56:15,015 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:56:15,015 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:56:15,015 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:56:15,015 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:56:15,016 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. 2023-04-25 02:56:15,016 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. started... 2023-04-25 02:56:15,016 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d6e38fead90b2a328d380bf5577ff0ce: 2023-04-25 02:56:15,016 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. 2023-04-25 02:56:15,017 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:56:15,017 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:56:15,017 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:15,017 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:56:15,017 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:56:15,019 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:15,019 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:15,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:15,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:15,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:15,020 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:15,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:15,020 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:56:15,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:15,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:15,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:15,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:15,021 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:56:15,021 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@28f757c1[Count = 0] remaining members to acquire global barrier 2023-04-25 02:56:15,021 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:56:15,021 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:15,022 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:15,022 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:15,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:15,022 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:56:15,022 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:56:15,022 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:56:15,022 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:15,023 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:56:15,024 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:56:15,024 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:56:15,024 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:56:15,024 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:15,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:15,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:15,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:15,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:15,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:15,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:15,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:15,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:15,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:15,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:15,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:56:15,026 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:56:15,026 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:56:15,027 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:56:15,027 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:56:15,027 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:56:15,027 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:56:15,028 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:15,028 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:56:15,028 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:15,028 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:15,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:15,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:15,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:15,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:15,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:15,029 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:56:15,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:56:15,029 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:15,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:15,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:15,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:15,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:15,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:15,031 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:15,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:15,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:15,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:15,034 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:56:15,034 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:56:15,034 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:56:15,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:56:15,034 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:15,034 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:56:15,034 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:56:15,034 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:56:15,034 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:15,035 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:56:15,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:56:15,035 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:15,035 DEBUG [Thread-3656] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:56:15,035 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:15,035 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:15,035 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:15,035 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:15,035 DEBUG [Thread-3656] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:56:15,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(9158): Flush requested on d6e38fead90b2a328d380bf5577ff0ce 2023-04-25 02:56:15,048 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing d6e38fead90b2a328d380bf5577ff0ce 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-25 02:56:15,049 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6e38fead90b2a328d380bf5577ff0ce, store=A 2023-04-25 02:56:15,049 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:56:15,050 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6e38fead90b2a328d380bf5577ff0ce, store=B 2023-04-25 02:56:15,050 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:56:15,050 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6e38fead90b2a328d380bf5577ff0ce, store=C 2023-04-25 02:56:15,050 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:56:15,063 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304252246c1fac0dd49a59575bfa6ae803c54 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304252246c1fac0dd49a59575bfa6ae803c54 2023-04-25 02:56:15,071 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=299, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/.tmp/A/9cab0e4d65854095a36511ae70a61873 2023-04-25 02:56:15,093 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:15,093 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:15,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39458 deadline: 1682391435089, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:15,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39526 deadline: 1682391435088, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:15,094 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682391375057, max timestamp=1682391375092Num uniques 859; 2023-04-25 02:56:15,094 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-25 02:56:15,094 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682391375057, max timestamp=1682391375092Num uniques 859; 2023-04-25 02:56:15,094 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682391375057, max timestamp=1682391375092Num uniques 859; 2023-04-25 02:56:15,094 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-25 02:56:15,094 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-25 02:56:15,116 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:15,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39482 deadline: 1682391435111, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:15,119 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:15,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39536 deadline: 1682391435117, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:15,121 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:15,121 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39522 deadline: 1682391435117, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:15,135 DEBUG [Thread-3656] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:56:15,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:56:15,139 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:56:15,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:56:15,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:56:15,141 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:56:15,141 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:56:15,141 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:56:15,141 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:56:15,141 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:15,141 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:15,142 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:15,142 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:56:15,143 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:56:15,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:56:15,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:15,143 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:15,143 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:56:15,143 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:15,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:56:15,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:15,145 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:56:15,148 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:56:15,148 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:56:15,148 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:56:15,148 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:56:15,148 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:56:15,156 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:56:15,156 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:56:15,156 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. 2023-04-25 02:56:15,156 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. started... 2023-04-25 02:56:15,156 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. as already flushing 2023-04-25 02:56:15,156 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. 2023-04-25 02:56:15,156 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:56:15,156 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:56:15,156 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:15,156 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:56:15,156 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:56:15,158 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:15,158 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:15,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:15,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:15,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:15,158 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:15,158 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:56:15,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:15,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:15,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:15,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:15,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:15,159 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:56:15,159 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7ce31cc[Count = 0] remaining members to acquire global barrier 2023-04-25 02:56:15,159 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:56:15,160 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:15,161 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:15,161 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:15,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:15,161 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:56:15,161 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:56:15,161 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:56:15,161 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:15,161 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:56:15,163 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:56:15,163 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:15,163 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:56:15,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:15,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:15,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:15,163 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:56:15,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:15,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:15,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:15,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:15,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:15,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:15,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:15,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:56:15,165 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:56:15,165 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:56:15,165 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:56:15,165 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:56:15,165 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:56:15,166 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:56:15,167 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:15,167 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:56:15,167 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:15,167 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:15,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:15,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:15,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:15,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:15,167 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:56:15,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:56:15,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:15,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:15,168 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:15,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:15,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:15,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:15,169 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:15,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:15,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:15,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:15,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:15,172 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:15,172 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:15,172 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:15,172 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:15,173 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:15,173 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:15,173 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:15,173 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:56:15,173 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:56:15,173 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:56:15,173 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:56:15,173 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:56:15,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:56:15,173 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:56:15,173 DEBUG [Thread-3656] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:56:15,173 DEBUG [Thread-3656] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:56:15,173 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:56:15,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:56:15,174 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:56:15,197 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:15,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39526 deadline: 1682391435195, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:15,197 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:15,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39458 deadline: 1682391435195, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:15,218 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:15,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39482 deadline: 1682391435218, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:15,222 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:15,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39536 deadline: 1682391435220, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:15,225 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:15,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39522 deadline: 1682391435223, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:15,273 DEBUG [Thread-3656] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:56:15,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:56:15,276 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:56:15,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:56:15,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:56:15,279 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:56:15,279 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:56:15,280 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:56:15,280 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:56:15,280 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:15,280 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:15,281 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:56:15,281 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:56:15,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:56:15,281 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:15,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:15,282 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:15,282 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:56:15,282 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:15,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:56:15,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:15,282 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:56:15,283 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:56:15,283 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:56:15,283 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:56:15,283 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:56:15,283 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:56:15,283 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:56:15,283 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:56:15,283 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. 2023-04-25 02:56:15,284 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. started... 2023-04-25 02:56:15,284 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. as already flushing 2023-04-25 02:56:15,284 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. 2023-04-25 02:56:15,284 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:56:15,284 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:56:15,284 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:15,284 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:56:15,284 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:56:15,286 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:15,286 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:15,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:15,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:15,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:15,287 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:15,287 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:56:15,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:15,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:15,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:15,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:15,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:15,288 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:56:15,288 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@262afa8e[Count = 0] remaining members to acquire global barrier 2023-04-25 02:56:15,288 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:56:15,288 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:15,289 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:15,289 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:15,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:15,289 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:56:15,289 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:56:15,289 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:56:15,290 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:15,290 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:56:15,291 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:15,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:15,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:15,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:15,291 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:56:15,291 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:56:15,291 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:56:15,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:15,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:15,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:15,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:15,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:15,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:15,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:15,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:56:15,293 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:56:15,293 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:56:15,293 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:56:15,293 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:56:15,293 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:56:15,293 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:56:15,295 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:15,295 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:15,295 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:15,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:15,295 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:56:15,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:15,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:15,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:15,295 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:56:15,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:56:15,295 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:15,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:15,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:15,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:15,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:15,296 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:15,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:15,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:15,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:15,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:15,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:15,302 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:56:15,302 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:56:15,302 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:56:15,302 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:56:15,302 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:56:15,302 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:56:15,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:56:15,302 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:15,302 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:56:15,302 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:15,302 DEBUG [Thread-3656] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:56:15,302 DEBUG [Thread-3656] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:56:15,302 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:15,302 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:56:15,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:56:15,302 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:15,303 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:15,303 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:15,303 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:15,400 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:15,401 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39458 deadline: 1682391435398, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:15,401 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:15,401 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39526 deadline: 1682391435399, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:15,402 DEBUG [Thread-3656] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:56:15,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:56:15,406 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:56:15,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:56:15,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:56:15,408 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:56:15,408 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:56:15,408 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:56:15,408 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:56:15,408 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:15,408 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:15,411 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:15,411 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:56:15,412 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:56:15,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:56:15,412 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:15,412 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:56:15,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:15,412 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:15,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:56:15,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:15,413 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:56:15,414 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:56:15,414 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:56:15,414 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:56:15,414 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:56:15,414 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:56:15,420 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:56:15,420 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:56:15,420 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. 2023-04-25 02:56:15,420 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. started... 2023-04-25 02:56:15,420 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. as already flushing 2023-04-25 02:56:15,420 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. 2023-04-25 02:56:15,420 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:56:15,420 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:56:15,420 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:15,420 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:56:15,420 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:56:15,422 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:15,422 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:15,422 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:15,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:15,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:15,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:15,422 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39482 deadline: 1682391435419, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:15,422 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:15,422 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:56:15,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:15,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:15,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:15,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:15,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:15,424 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:56:15,424 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3fe1b0d3[Count = 0] remaining members to acquire global barrier 2023-04-25 02:56:15,424 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:56:15,424 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:15,426 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:15,426 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:15,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:15,426 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:56:15,426 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:56:15,426 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:56:15,426 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:15,426 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:56:15,427 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:15,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39536 deadline: 1682391435424, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:15,429 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:15,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39522 deadline: 1682391435427, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:15,431 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:15,431 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:56:15,431 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:56:15,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:15,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:15,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:15,431 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:56:15,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:15,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:15,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:15,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:15,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:15,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:15,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:15,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:56:15,434 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:56:15,434 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:56:15,434 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:56:15,434 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:56:15,434 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:56:15,434 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:56:15,436 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:15,436 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:15,436 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:56:15,436 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:15,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:15,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:15,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:15,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:15,436 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:56:15,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:56:15,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:15,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:15,437 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:15,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:15,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:15,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:15,438 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:15,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:15,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:15,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:15,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:15,441 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:56:15,441 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:56:15,441 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:15,441 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:15,441 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:15,441 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:15,441 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:56:15,441 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:56:15,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:56:15,441 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:56:15,441 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:56:15,441 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:56:15,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:56:15,441 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:15,441 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:15,441 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:56:15,441 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:15,441 DEBUG [Thread-3656] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:56:15,442 DEBUG [Thread-3656] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:56:15,484 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=299 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/.tmp/B/9e67dc4933b8425fa704a29c92b141bf 2023-04-25 02:56:15,493 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=299 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/.tmp/C/509e90ab9f7f44eabc903244a132da5d 2023-04-25 02:56:15,497 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/.tmp/A/9cab0e4d65854095a36511ae70a61873 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/A/9cab0e4d65854095a36511ae70a61873 2023-04-25 02:56:15,500 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/A/9cab0e4d65854095a36511ae70a61873, entries=150, sequenceid=299, filesize=25.4 K 2023-04-25 02:56:15,501 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/.tmp/B/9e67dc4933b8425fa704a29c92b141bf as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/B/9e67dc4933b8425fa704a29c92b141bf 2023-04-25 02:56:15,503 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/B/9e67dc4933b8425fa704a29c92b141bf, entries=150, sequenceid=299, filesize=11.9 K 2023-04-25 02:56:15,504 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/.tmp/C/509e90ab9f7f44eabc903244a132da5d as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/C/509e90ab9f7f44eabc903244a132da5d 2023-04-25 02:56:15,508 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/C/509e90ab9f7f44eabc903244a132da5d, entries=150, sequenceid=299, filesize=11.9 K 2023-04-25 02:56:15,508 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=134.18 KB/137400 for d6e38fead90b2a328d380bf5577ff0ce in 460ms, sequenceid=299, compaction requested=true 2023-04-25 02:56:15,508 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for d6e38fead90b2a328d380bf5577ff0ce: 2023-04-25 02:56:15,508 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 02:56:15,508 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-25 02:56:15,509 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-25 02:56:15,509 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 02:56:15,509 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 02:56:15,510 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 78500 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-25 02:56:15,510 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HStore(1912): d6e38fead90b2a328d380bf5577ff0ce/A is initiating minor compaction (all files) 2023-04-25 02:56:15,510 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of d6e38fead90b2a328d380bf5577ff0ce/A in TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. 2023-04-25 02:56:15,510 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36920 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-25 02:56:15,510 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/A/435557a72796411297823d2e5692300e, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/A/1d6084fa00f646f4b04857d7ae1bcf77, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/A/9cab0e4d65854095a36511ae70a61873] into tmpdir=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/.tmp, totalSize=76.7 K 2023-04-25 02:56:15,510 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HStore(1912): d6e38fead90b2a328d380bf5577ff0ce/B is initiating minor compaction (all files) 2023-04-25 02:56:15,510 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d6e38fead90b2a328d380bf5577ff0ce/B in TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. 2023-04-25 02:56:15,510 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/B/9362dadfefb742f7a473b4f8e3e5c47a, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/B/3ce395b9822f4c0da488ea66c3782d9c, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/B/9e67dc4933b8425fa704a29c92b141bf] into tmpdir=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/.tmp, totalSize=36.1 K 2023-04-25 02:56:15,510 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting 435557a72796411297823d2e5692300e, keycount=150, bloomtype=ROW, size=25.8 K, encoding=NONE, compression=NONE, seqNum=229, earliestPutTs=1682391372799 2023-04-25 02:56:15,511 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting 9362dadfefb742f7a473b4f8e3e5c47a, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=229, earliestPutTs=1682391372799 2023-04-25 02:56:15,511 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting 1d6084fa00f646f4b04857d7ae1bcf77, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=283, earliestPutTs=1682391372903 2023-04-25 02:56:15,511 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting 3ce395b9822f4c0da488ea66c3782d9c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=283, earliestPutTs=1682391372903 2023-04-25 02:56:15,511 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting 9cab0e4d65854095a36511ae70a61873, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=299, earliestPutTs=1682391375046 2023-04-25 02:56:15,511 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting 9e67dc4933b8425fa704a29c92b141bf, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=299, earliestPutTs=1682391375046 2023-04-25 02:56:15,520 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] throttle.PressureAwareThroughputController(145): d6e38fead90b2a328d380bf5577ff0ce#B#compaction#538 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 53.85 MB/second 2023-04-25 02:56:15,520 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] throttle.PressureAwareThroughputController(145): d6e38fead90b2a328d380bf5577ff0ce#A#compaction#537 average throughput is 4.93 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-25 02:56:15,542 DEBUG [Thread-3656] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:56:15,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:56:15,545 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:56:15,546 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/.tmp/B/0cc319688f0c49749f96a064e514af7f as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/B/0cc319688f0c49749f96a064e514af7f 2023-04-25 02:56:15,550 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in d6e38fead90b2a328d380bf5577ff0ce/B of d6e38fead90b2a328d380bf5577ff0ce into 0cc319688f0c49749f96a064e514af7f(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 02:56:15,551 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d6e38fead90b2a328d380bf5577ff0ce: 2023-04-25 02:56:15,551 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce., storeName=d6e38fead90b2a328d380bf5577ff0ce/B, priority=13, startTime=1682391375509; duration=0sec 2023-04-25 02:56:15,551 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 02:56:15,551 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-25 02:56:15,552 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36920 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-25 02:56:15,552 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HStore(1912): d6e38fead90b2a328d380bf5577ff0ce/C is initiating minor compaction (all files) 2023-04-25 02:56:15,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:56:15,552 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d6e38fead90b2a328d380bf5577ff0ce/C in TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. 2023-04-25 02:56:15,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:56:15,552 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/C/0fcbff231ef140a79b687e50fc9b047d, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/C/e275a85e2bf44fd19839f67df8bfb4d8, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/C/509e90ab9f7f44eabc903244a132da5d] into tmpdir=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/.tmp, totalSize=36.1 K 2023-04-25 02:56:15,552 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:56:15,552 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:56:15,552 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:56:15,552 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:56:15,552 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting 0fcbff231ef140a79b687e50fc9b047d, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=229, earliestPutTs=1682391372799 2023-04-25 02:56:15,553 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:15,553 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:15,553 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting e275a85e2bf44fd19839f67df8bfb4d8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=283, earliestPutTs=1682391372903 2023-04-25 02:56:15,553 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting 509e90ab9f7f44eabc903244a132da5d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=299, earliestPutTs=1682391375046 2023-04-25 02:56:15,554 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:56:15,554 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:15,554 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:56:15,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:56:15,554 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:15,554 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:56:15,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:15,554 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:15,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:56:15,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:15,555 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:56:15,556 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:56:15,556 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:56:15,556 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:56:15,556 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:56:15,556 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:56:15,556 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:56:15,556 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:56:15,556 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. 2023-04-25 02:56:15,557 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. started... 2023-04-25 02:56:15,557 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d6e38fead90b2a328d380bf5577ff0ce 3/3 column families, dataSize=134.18 KB heapSize=380.30 KB 2023-04-25 02:56:15,557 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6e38fead90b2a328d380bf5577ff0ce, store=A 2023-04-25 02:56:15,557 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 02:56:15,557 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6e38fead90b2a328d380bf5577ff0ce, store=B 2023-04-25 02:56:15,557 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 02:56:15,557 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6e38fead90b2a328d380bf5577ff0ce, store=C 2023-04-25 02:56:15,557 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 02:56:15,558 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] throttle.PressureAwareThroughputController(145): d6e38fead90b2a328d380bf5577ff0ce#C#compaction#539 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-25 02:56:15,568 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/.tmp/C/348fde98b0f1414e90d760f01955a23a as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/C/348fde98b0f1414e90d760f01955a23a 2023-04-25 02:56:15,573 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in d6e38fead90b2a328d380bf5577ff0ce/C of d6e38fead90b2a328d380bf5577ff0ce into 348fde98b0f1414e90d760f01955a23a(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 02:56:15,573 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d6e38fead90b2a328d380bf5577ff0ce: 2023-04-25 02:56:15,573 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce., storeName=d6e38fead90b2a328d380bf5577ff0ce/C, priority=13, startTime=1682391375509; duration=0sec 2023-04-25 02:56:15,573 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 02:56:15,578 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/.tmp/A/6954056042cb40a78c6c5f586eb576a7 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/A/6954056042cb40a78c6c5f586eb576a7 2023-04-25 02:56:15,582 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in d6e38fead90b2a328d380bf5577ff0ce/A of d6e38fead90b2a328d380bf5577ff0ce into 6954056042cb40a78c6c5f586eb576a7(size=26.0 K), total size for store is 26.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 02:56:15,582 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for d6e38fead90b2a328d380bf5577ff0ce: 2023-04-25 02:56:15,582 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce., storeName=d6e38fead90b2a328d380bf5577ff0ce/A, priority=13, startTime=1682391375508; duration=0sec 2023-04-25 02:56:15,582 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 02:56:15,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(9158): Flush requested on d6e38fead90b2a328d380bf5577ff0ce 2023-04-25 02:56:15,705 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. as already flushing 2023-04-25 02:56:15,720 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:15,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39458 deadline: 1682391435717, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:15,720 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:15,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39526 deadline: 1682391435717, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:15,728 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:15,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39482 deadline: 1682391435725, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:15,732 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:15,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39536 deadline: 1682391435730, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:15,732 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:15,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39522 deadline: 1682391435731, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:15,821 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:15,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39458 deadline: 1682391435821, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:15,822 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:15,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39526 deadline: 1682391435821, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:15,978 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230425d8b9a92634c24331a7bfe34a6fa330b4 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425d8b9a92634c24331a7bfe34a6fa330b4 2023-04-25 02:56:15,983 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=336, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/.tmp/A/56a6c8b8d04444d1b77062034f787a62 2023-04-25 02:56:16,026 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:16,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39526 deadline: 1682391436023, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:16,026 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:16,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39458 deadline: 1682391436024, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:16,230 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:16,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39482 deadline: 1682391436229, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:16,234 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:16,234 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39522 deadline: 1682391436233, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:16,237 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:16,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39536 deadline: 1682391436237, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:16,330 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:16,331 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39526 deadline: 1682391436329, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:16,331 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:16,331 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39458 deadline: 1682391436329, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:16,392 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=336 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/.tmp/B/e5632252158d4d1987c1fc9973c5a9a8 2023-04-25 02:56:16,401 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=336 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/.tmp/C/40899778f8fc4d7da81c2205efe78c6b 2023-04-25 02:56:16,405 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/.tmp/A/56a6c8b8d04444d1b77062034f787a62 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/A/56a6c8b8d04444d1b77062034f787a62 2023-04-25 02:56:16,408 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/A/56a6c8b8d04444d1b77062034f787a62, entries=150, sequenceid=336, filesize=25.4 K 2023-04-25 02:56:16,409 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/.tmp/B/e5632252158d4d1987c1fc9973c5a9a8 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/B/e5632252158d4d1987c1fc9973c5a9a8 2023-04-25 02:56:16,414 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/B/e5632252158d4d1987c1fc9973c5a9a8, entries=150, sequenceid=336, filesize=11.9 K 2023-04-25 02:56:16,414 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/.tmp/C/40899778f8fc4d7da81c2205efe78c6b as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/C/40899778f8fc4d7da81c2205efe78c6b 2023-04-25 02:56:16,419 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/C/40899778f8fc4d7da81c2205efe78c6b, entries=150, sequenceid=336, filesize=11.9 K 2023-04-25 02:56:16,420 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~380.25 KB/389376, currentSize=53.67 KB/54960 for d6e38fead90b2a328d380bf5577ff0ce in 863ms, sequenceid=336, compaction requested=false 2023-04-25 02:56:16,420 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d6e38fead90b2a328d380bf5577ff0ce: 2023-04-25 02:56:16,420 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. 2023-04-25 02:56:16,420 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:56:16,420 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:56:16,420 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:16,420 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:56:16,420 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:56:16,423 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:16,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:16,423 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:16,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:16,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:16,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:16,424 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:16,424 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:56:16,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:16,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:16,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:16,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:16,425 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:56:16,425 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2dbaa1b1[Count = 0] remaining members to acquire global barrier 2023-04-25 02:56:16,425 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:56:16,426 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:16,426 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:16,427 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:16,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:16,427 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:56:16,427 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:56:16,427 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:56:16,427 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:16,427 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:56:16,428 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:16,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:16,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:16,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:16,428 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:56:16,428 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:56:16,428 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:56:16,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:16,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:16,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:16,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:16,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:16,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:16,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:16,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:56:16,431 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:56:16,431 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:56:16,431 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:56:16,431 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:56:16,431 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:56:16,431 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:56:16,432 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:16,432 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:16,432 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:56:16,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:16,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:16,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:16,432 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:16,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:16,433 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:56:16,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:56:16,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:16,433 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:16,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:16,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:16,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:16,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:16,434 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:16,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:16,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:16,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:16,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:16,438 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:16,438 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:16,438 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:56:16,438 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:56:16,438 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:56:16,438 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:56:16,438 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:56:16,438 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:56:16,438 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:56:16,438 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:16,438 DEBUG [Thread-3656] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:56:16,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:56:16,438 DEBUG [Thread-3656] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:56:16,438 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:16,438 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:16,438 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:16,438 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:16,438 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:56:16,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:56:16,538 DEBUG [Thread-3656] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:56:16,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:56:16,540 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:56:16,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:56:16,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:56:16,542 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:56:16,542 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:56:16,542 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:56:16,542 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:56:16,542 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:16,542 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:16,543 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:56:16,543 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:16,543 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:56:16,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:56:16,544 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:16,544 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:56:16,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:16,544 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:16,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:56:16,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:16,545 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:56:16,545 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:56:16,545 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:56:16,545 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:56:16,545 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:56:16,545 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:56:16,545 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:56:16,545 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. 2023-04-25 02:56:16,545 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. started... 2023-04-25 02:56:16,545 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:56:16,546 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d6e38fead90b2a328d380bf5577ff0ce 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-25 02:56:16,546 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6e38fead90b2a328d380bf5577ff0ce, store=A 2023-04-25 02:56:16,546 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:56:16,546 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6e38fead90b2a328d380bf5577ff0ce, store=B 2023-04-25 02:56:16,546 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:56:16,546 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6e38fead90b2a328d380bf5577ff0ce, store=C 2023-04-25 02:56:16,546 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:56:16,568 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230425041d8f01945e44ca84d0202721ed0aed to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425041d8f01945e44ca84d0202721ed0aed 2023-04-25 02:56:16,574 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=349, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/.tmp/A/2c4f81230cd1496899f70b9fcfd2d61f 2023-04-25 02:56:16,583 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=349 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/.tmp/B/2fe830a5f0ca46d097b6ad9d1c9e7380 2023-04-25 02:56:16,595 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=349 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/.tmp/C/29c6956aa568405a8b37a6c997cfe5fb 2023-04-25 02:56:16,598 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/.tmp/A/2c4f81230cd1496899f70b9fcfd2d61f as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/A/2c4f81230cd1496899f70b9fcfd2d61f 2023-04-25 02:56:16,602 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/A/2c4f81230cd1496899f70b9fcfd2d61f, entries=100, sequenceid=349, filesize=18.6 K 2023-04-25 02:56:16,602 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/.tmp/B/2fe830a5f0ca46d097b6ad9d1c9e7380 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/B/2fe830a5f0ca46d097b6ad9d1c9e7380 2023-04-25 02:56:16,606 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/B/2fe830a5f0ca46d097b6ad9d1c9e7380, entries=100, sequenceid=349, filesize=9.5 K 2023-04-25 02:56:16,607 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/.tmp/C/29c6956aa568405a8b37a6c997cfe5fb as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/C/29c6956aa568405a8b37a6c997cfe5fb 2023-04-25 02:56:16,610 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/C/29c6956aa568405a8b37a6c997cfe5fb, entries=100, sequenceid=349, filesize=9.5 K 2023-04-25 02:56:16,611 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=0 B/0 for d6e38fead90b2a328d380bf5577ff0ce in 65ms, sequenceid=349, compaction requested=true 2023-04-25 02:56:16,611 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d6e38fead90b2a328d380bf5577ff0ce: 2023-04-25 02:56:16,611 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. 2023-04-25 02:56:16,611 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:56:16,611 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:56:16,611 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:16,611 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:56:16,611 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:56:16,612 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:16,613 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:16,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:16,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:16,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:16,613 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:16,613 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:56:16,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:16,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:16,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:16,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:16,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:16,614 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:56:16,615 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5b0cb59b[Count = 0] remaining members to acquire global barrier 2023-04-25 02:56:16,615 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:56:16,615 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:16,616 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:16,616 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:16,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:16,616 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:56:16,616 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:56:16,616 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:16,616 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:56:16,616 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:56:16,619 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:56:16,619 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:16,619 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:56:16,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:16,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:16,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:16,619 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:56:16,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:16,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:16,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:16,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:16,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:16,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:16,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:16,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:56:16,622 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:56:16,622 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:56:16,622 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:56:16,622 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:56:16,622 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:56:16,622 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:56:16,623 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:16,623 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:16,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:16,624 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:16,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:16,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:16,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:16,623 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:56:16,624 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:16,624 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:56:16,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:56:16,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:16,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:16,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:16,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:16,625 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:16,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:16,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:16,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:16,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:16,627 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:16,627 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:16,627 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:16,627 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:56:16,627 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:56:16,627 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:16,627 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:56:16,628 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:56:16,628 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:56:16,628 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:16,628 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:56:16,628 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:56:16,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:56:16,628 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:16,628 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:16,628 DEBUG [Thread-3656] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:56:16,628 DEBUG [Thread-3656] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:56:16,628 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:56:16,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:56:16,728 DEBUG [Thread-3656] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:56:16,748 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:56:16,750 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:56:16,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:56:16,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:56:16,752 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:56:16,752 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:56:16,752 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:56:16,752 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:56:16,752 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:16,752 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:16,754 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:56:16,754 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:16,754 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:56:16,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:56:16,754 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:16,754 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:56:16,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:16,755 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:16,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:56:16,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:16,755 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:56:16,755 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:56:16,755 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:56:16,755 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:56:16,755 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:56:16,756 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:56:16,756 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. 2023-04-25 02:56:16,756 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. started... 2023-04-25 02:56:16,756 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:56:16,756 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d6e38fead90b2a328d380bf5577ff0ce: 2023-04-25 02:56:16,756 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. 2023-04-25 02:56:16,756 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:56:16,756 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:56:16,756 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:16,756 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:56:16,756 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:56:16,757 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:16,757 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:16,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:16,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:16,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:16,758 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:16,758 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:56:16,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:16,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:16,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:16,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:16,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:16,759 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:56:16,760 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2a03dafe[Count = 0] remaining members to acquire global barrier 2023-04-25 02:56:16,760 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:56:16,760 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:16,761 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:16,761 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:16,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:16,761 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:56:16,761 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:56:16,761 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:56:16,761 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:16,761 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:56:16,762 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:16,762 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:56:16,762 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:56:16,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:16,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:16,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:16,762 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:56:16,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:16,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:16,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:16,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:16,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:16,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:16,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:16,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:56:16,765 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:56:16,765 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:56:16,765 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:56:16,765 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:56:16,765 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:56:16,765 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:56:16,767 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:16,767 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:16,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:16,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:16,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:16,767 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:56:16,767 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:16,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:16,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:16,768 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:56:16,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:56:16,768 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:16,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:16,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:16,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:16,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:16,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:16,769 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:16,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:16,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:16,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:16,772 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:16,772 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:56:16,772 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:16,772 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:56:16,772 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:56:16,772 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:56:16,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:56:16,772 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:56:16,772 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:16,772 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:56:16,772 DEBUG [Thread-3656] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:56:16,772 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:16,772 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:16,772 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:56:16,772 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:56:16,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:56:16,772 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:16,773 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:16,772 DEBUG [Thread-3656] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:56:16,873 DEBUG [Thread-3656] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:56:16,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:56:16,875 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:56:16,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:56:16,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:56:16,879 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:56:16,879 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:56:16,879 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:56:16,879 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:56:16,879 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:16,879 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:16,880 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:16,880 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:56:16,880 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:56:16,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:56:16,881 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:16,881 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:56:16,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:16,882 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:16,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:56:16,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:16,882 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:56:16,883 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:56:16,883 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:56:16,883 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:56:16,883 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:56:16,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(9158): Flush requested on d6e38fead90b2a328d380bf5577ff0ce 2023-04-25 02:56:16,883 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing d6e38fead90b2a328d380bf5577ff0ce 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-25 02:56:16,883 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:56:16,883 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6e38fead90b2a328d380bf5577ff0ce, store=A 2023-04-25 02:56:16,883 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:56:16,883 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:56:16,883 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:56:16,883 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. 2023-04-25 02:56:16,883 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. started... 2023-04-25 02:56:16,883 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6e38fead90b2a328d380bf5577ff0ce, store=B 2023-04-25 02:56:16,884 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. as already flushing 2023-04-25 02:56:16,884 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:56:16,884 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. 2023-04-25 02:56:16,884 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6e38fead90b2a328d380bf5577ff0ce, store=C 2023-04-25 02:56:16,884 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:56:16,884 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:56:16,884 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:16,884 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:56:16,884 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:56:16,884 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:56:16,886 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:16,886 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:16,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:16,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:16,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:16,888 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:16,888 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:56:16,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:16,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:16,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:16,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:16,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:16,893 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:56:16,893 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@18569045[Count = 0] remaining members to acquire global barrier 2023-04-25 02:56:16,893 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:56:16,893 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:16,895 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:16,895 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:16,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:16,895 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:16,895 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:56:16,896 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:56:16,895 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:56:16,896 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:56:16,897 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:56:16,897 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:56:16,897 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:16,897 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:56:16,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:16,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:16,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:16,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:16,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:16,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:16,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:16,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:16,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:16,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:16,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:56:16,899 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:56:16,899 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:56:16,899 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:56:16,899 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:56:16,899 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:56:16,899 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:56:16,901 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:16,901 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:16,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:16,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:16,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:16,901 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:16,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:16,901 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:56:16,901 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:16,901 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:56:16,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:56:16,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:16,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:16,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:16,902 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:16,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:16,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:16,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:16,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:16,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:16,904 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:56:16,905 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:56:16,905 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:16,905 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:56:16,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:56:16,905 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:56:16,905 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:56:16,905 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:56:16,905 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:16,905 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:16,905 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:16,905 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:16,905 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:16,905 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:16,905 DEBUG [Thread-3656] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:56:16,905 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:56:16,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:56:16,905 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:56:16,905 DEBUG [Thread-3656] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:56:16,940 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682391376927, max timestamp=1682391376938Num uniques 859; 2023-04-25 02:56:16,940 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682391376927, max timestamp=1682391376938Num uniques 859; 2023-04-25 02:56:16,940 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-25 02:56:16,940 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-25 02:56:16,942 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1682391376927, max timestamp=1682391376938Num uniques 859; 2023-04-25 02:56:16,942 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-04-25 02:56:16,976 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1682391376959, max timestamp=1682391376972Num uniques 1018; 2023-04-25 02:56:16,976 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1682391376959, max timestamp=1682391376972Num uniques 1009; 2023-04-25 02:56:16,976 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1682391376959, max timestamp=1682391376972Num uniques 1018; 2023-04-25 02:56:16,976 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=46213, new segment data size=9978, suffix heap size=132904,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=218 2023-04-25 02:56:16,976 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-25 02:56:16,976 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-25 02:56:17,005 DEBUG [Thread-3656] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:56:17,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:56:17,007 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:56:17,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:56:17,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:56:17,009 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:56:17,009 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:56:17,009 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:56:17,009 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:56:17,009 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:17,009 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:17,010 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:17,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39526 deadline: 1682391437006, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:17,010 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:17,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39458 deadline: 1682391437007, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:17,011 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:56:17,011 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:17,011 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:56:17,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:56:17,012 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:17,012 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:56:17,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:17,012 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:17,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:56:17,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:17,012 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:56:17,013 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:56:17,013 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:56:17,013 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:56:17,013 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:56:17,013 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:56:17,013 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:56:17,013 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:56:17,013 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. 2023-04-25 02:56:17,013 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. started... 2023-04-25 02:56:17,013 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. as already flushing 2023-04-25 02:56:17,013 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. 2023-04-25 02:56:17,014 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:56:17,014 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:56:17,014 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:17,014 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:56:17,014 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:56:17,015 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:17,015 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:17,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:17,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:17,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:17,015 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:17,015 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:56:17,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:17,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:17,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:17,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:17,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:17,017 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:56:17,017 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@60190dd7[Count = 0] remaining members to acquire global barrier 2023-04-25 02:56:17,017 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:56:17,017 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:17,018 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:17,018 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:17,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:17,018 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:56:17,018 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:56:17,018 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:17,018 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:56:17,018 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:56:17,020 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:17,020 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:56:17,020 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:56:17,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:17,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:17,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:17,020 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:56:17,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:17,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:17,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:17,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:17,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:17,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:17,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:17,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:56:17,022 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:56:17,022 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:56:17,022 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:56:17,022 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:56:17,022 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:56:17,022 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:56:17,024 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:17,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:17,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:17,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:17,025 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:17,025 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:56:17,025 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:17,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:17,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:17,026 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:56:17,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:56:17,026 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:17,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:17,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:17,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:17,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:17,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:17,027 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:17,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:17,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:17,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:17,029 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:56:17,029 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:56:17,029 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:56:17,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:56:17,029 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:56:17,029 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:56:17,029 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:56:17,029 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:17,030 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:17,030 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:17,030 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:17,030 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:17,030 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:17,030 DEBUG [Thread-3656] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:56:17,030 DEBUG [Thread-3656] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:56:17,030 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:56:17,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:56:17,030 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:17,113 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:17,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39526 deadline: 1682391437111, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:17,113 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:17,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39458 deadline: 1682391437111, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:17,130 DEBUG [Thread-3656] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:56:17,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:56:17,132 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:56:17,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:56:17,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:56:17,133 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:56:17,133 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:56:17,133 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:56:17,133 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:56:17,133 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:17,133 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:17,137 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:17,138 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:56:17,138 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:56:17,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:56:17,138 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:17,138 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:56:17,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:17,138 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:17,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:56:17,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:17,138 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:56:17,139 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:56:17,139 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:56:17,139 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:56:17,139 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:56:17,139 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:56:17,139 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:56:17,139 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:56:17,139 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. 2023-04-25 02:56:17,139 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. started... 2023-04-25 02:56:17,139 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. as already flushing 2023-04-25 02:56:17,139 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. 2023-04-25 02:56:17,139 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:56:17,139 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:56:17,139 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:17,139 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:56:17,139 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:56:17,141 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:17,141 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:17,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:17,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:17,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:17,141 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:17,141 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:56:17,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:17,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:17,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:17,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:17,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:17,142 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:56:17,142 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5e07dc22[Count = 0] remaining members to acquire global barrier 2023-04-25 02:56:17,142 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:56:17,142 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:17,143 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:17,143 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:17,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:17,143 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:56:17,143 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:56:17,143 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:56:17,144 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:17,144 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:56:17,148 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:17,148 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:56:17,148 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:56:17,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:17,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:17,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:17,148 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:56:17,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:17,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:17,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:17,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:17,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:17,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:17,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:17,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:56:17,150 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:56:17,150 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:56:17,150 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:56:17,150 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:56:17,150 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:56:17,150 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:56:17,151 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:17,151 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:17,151 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:17,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:17,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:17,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:17,151 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:56:17,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:17,152 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:56:17,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:56:17,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:17,152 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:17,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:17,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:17,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:17,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:17,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:17,153 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:17,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:17,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:17,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:17,155 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:17,155 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:17,155 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:17,155 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:56:17,155 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:56:17,155 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:56:17,155 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:17,156 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:56:17,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:56:17,155 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:56:17,156 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:56:17,155 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:56:17,156 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:17,156 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:17,156 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:17,156 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:56:17,156 DEBUG [Thread-3656] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:56:17,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:56:17,156 DEBUG [Thread-3656] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:56:17,233 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:17,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39482 deadline: 1682391437231, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:17,239 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:17,239 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39522 deadline: 1682391437239, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:17,242 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:17,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39536 deadline: 1682391437240, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:17,256 DEBUG [Thread-3656] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:56:17,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:56:17,258 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:56:17,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:56:17,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:56:17,260 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:56:17,260 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:56:17,260 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:56:17,260 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:56:17,260 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:17,260 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:17,261 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:17,261 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:56:17,261 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:56:17,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:56:17,262 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:17,262 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:56:17,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:17,262 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:17,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:56:17,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:17,263 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:56:17,263 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:56:17,263 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:56:17,263 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:56:17,263 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:56:17,263 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:56:17,263 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:56:17,263 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:56:17,263 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. 2023-04-25 02:56:17,263 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. started... 2023-04-25 02:56:17,264 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. as already flushing 2023-04-25 02:56:17,264 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. 2023-04-25 02:56:17,264 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:56:17,264 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:56:17,264 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:17,264 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:56:17,264 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:56:17,266 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:17,266 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:17,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:17,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:17,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:17,266 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:17,267 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:56:17,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:17,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:17,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:17,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:17,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:17,268 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:56:17,269 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6ae72936[Count = 0] remaining members to acquire global barrier 2023-04-25 02:56:17,269 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:56:17,269 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:17,270 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:17,270 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:17,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:17,270 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:56:17,270 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:56:17,270 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:56:17,270 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:17,270 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:56:17,271 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:17,271 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:56:17,271 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:56:17,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:17,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:17,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:17,271 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:56:17,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:17,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:17,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:17,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:17,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:17,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:17,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:17,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:56:17,274 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:56:17,274 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:56:17,274 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:56:17,275 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:56:17,275 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:56:17,275 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:56:17,276 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:17,276 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:56:17,276 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:17,276 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:17,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:17,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:17,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:17,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:17,277 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:56:17,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:56:17,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:17,277 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:17,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:17,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:17,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:17,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:17,278 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:17,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:17,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:17,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:17,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:17,282 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:56:17,282 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:56:17,282 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:56:17,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:56:17,282 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:56:17,282 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:17,282 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:17,282 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:17,282 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:17,282 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:17,282 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:17,282 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:17,282 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:56:17,282 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:56:17,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:56:17,282 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:56:17,282 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:56:17,282 DEBUG [Thread-3656] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:56:17,283 DEBUG [Thread-3656] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:56:17,306 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304251f7f8b2b5bb549b5b722167d144e0d93 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304251f7f8b2b5bb549b5b722167d144e0d93 2023-04-25 02:56:17,310 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=360, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/.tmp/A/1878815b560446ac8dda696440dbddfb 2023-04-25 02:56:17,318 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:17,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39458 deadline: 1682391437316, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:17,322 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:17,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39526 deadline: 1682391437320, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:17,331 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=360 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/.tmp/B/ac5006cce1e9465db77996c879df7734 2023-04-25 02:56:17,383 DEBUG [Thread-3656] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:56:17,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:56:17,385 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:56:17,386 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:56:17,386 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:56:17,386 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:56:17,386 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:56:17,386 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:56:17,386 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:56:17,387 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:17,387 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:17,388 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:17,388 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:56:17,388 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:56:17,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:56:17,388 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:17,388 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:56:17,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:17,389 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:17,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:56:17,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:17,389 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:56:17,389 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:56:17,389 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:56:17,389 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:56:17,389 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:56:17,389 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:56:17,390 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:56:17,390 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:56:17,390 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. 2023-04-25 02:56:17,390 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. started... 2023-04-25 02:56:17,390 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. as already flushing 2023-04-25 02:56:17,390 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. 2023-04-25 02:56:17,390 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:56:17,390 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:56:17,390 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:17,390 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:56:17,390 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:56:17,391 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:17,391 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:17,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:17,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:17,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:17,392 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:17,392 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:56:17,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:17,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:17,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:17,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:17,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:17,393 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:56:17,393 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@469f4897[Count = 0] remaining members to acquire global barrier 2023-04-25 02:56:17,393 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:56:17,393 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:17,394 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:17,394 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:17,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:17,394 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:56:17,394 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:56:17,394 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:56:17,394 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:17,394 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:56:17,396 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:17,396 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:56:17,396 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:56:17,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:17,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:17,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:17,396 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:56:17,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:17,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:17,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:17,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:17,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:17,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:17,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:17,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:56:17,399 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:56:17,399 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:56:17,399 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:56:17,399 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:56:17,399 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:56:17,399 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:56:17,400 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:17,400 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:17,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:17,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:17,400 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:17,400 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:56:17,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:17,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:17,401 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:56:17,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:56:17,401 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:17,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:17,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:17,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:17,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:17,402 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:17,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:17,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:17,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:17,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:17,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:17,404 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:56:17,404 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:17,404 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:56:17,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:56:17,404 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:17,404 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:56:17,404 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:56:17,404 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:56:17,405 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:56:17,404 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:17,405 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:17,405 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:17,405 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:17,405 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:17,405 DEBUG [Thread-3656] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:56:17,405 DEBUG [Thread-3656] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:56:17,405 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:56:17,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:56:17,406 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:56:17,505 DEBUG [Thread-3656] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:56:17,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:56:17,507 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:56:17,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:56:17,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:56:17,508 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:56:17,508 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:56:17,508 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:56:17,508 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:56:17,509 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:17,509 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:17,510 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:56:17,510 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:17,510 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:56:17,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:56:17,510 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:17,510 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:56:17,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:17,511 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:17,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:56:17,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:17,511 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:56:17,511 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:56:17,511 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:56:17,511 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:56:17,511 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:56:17,511 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:56:17,511 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:56:17,512 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:56:17,512 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. 2023-04-25 02:56:17,512 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. started... 2023-04-25 02:56:17,512 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. as already flushing 2023-04-25 02:56:17,512 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. 2023-04-25 02:56:17,512 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:56:17,512 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:56:17,512 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:17,512 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:56:17,512 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:56:17,514 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:17,514 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:17,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:17,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:17,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:17,516 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:17,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:17,516 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:56:17,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:17,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:17,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:17,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:17,518 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:56:17,518 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6ec09226[Count = 0] remaining members to acquire global barrier 2023-04-25 02:56:17,518 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:56:17,518 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:17,521 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:17,521 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:17,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:17,522 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:56:17,522 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:56:17,522 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:17,522 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:56:17,522 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:56:17,523 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:17,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:17,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:17,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:17,523 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:56:17,523 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:56:17,523 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:56:17,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:17,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:17,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:17,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:17,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:17,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:17,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:17,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:56:17,526 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:56:17,526 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:56:17,526 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:56:17,526 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:56:17,526 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:56:17,526 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:56:17,528 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:17,528 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:56:17,528 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:17,528 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:17,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:17,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:17,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:17,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:17,528 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:56:17,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:56:17,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:17,528 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:17,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:17,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:17,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:17,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:17,530 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:17,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:17,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:17,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:17,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:17,534 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:56:17,534 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:56:17,534 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:56:17,534 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:56:17,534 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:56:17,534 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:56:17,534 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:56:17,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:56:17,534 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:17,534 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:17,535 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:17,535 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:17,535 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:17,535 DEBUG [Thread-3656] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:56:17,535 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:17,535 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:56:17,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:56:17,535 DEBUG [Thread-3656] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:56:17,535 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:17,621 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:17,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39458 deadline: 1682391437620, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:17,627 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:17,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39526 deadline: 1682391437625, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:17,635 DEBUG [Thread-3656] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:56:17,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:56:17,637 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:56:17,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:56:17,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:56:17,638 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:56:17,638 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:56:17,638 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:56:17,638 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:56:17,639 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:17,639 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:17,640 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:17,640 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:56:17,640 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:56:17,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:56:17,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:17,641 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:17,641 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:56:17,641 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:17,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:56:17,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:17,642 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:56:17,642 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:56:17,642 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:56:17,642 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:56:17,642 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:56:17,642 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:56:17,642 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:56:17,643 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:56:17,643 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. 2023-04-25 02:56:17,643 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. started... 2023-04-25 02:56:17,643 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. as already flushing 2023-04-25 02:56:17,643 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. 2023-04-25 02:56:17,643 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:56:17,643 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:56:17,643 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:17,643 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:56:17,643 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:56:17,644 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:17,644 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:17,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:17,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:17,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:17,645 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:17,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:17,645 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:56:17,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:17,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:17,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:17,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:17,647 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:56:17,647 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@e46a510[Count = 0] remaining members to acquire global barrier 2023-04-25 02:56:17,647 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:56:17,647 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:17,648 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:17,648 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:17,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:17,649 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:56:17,649 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:56:17,649 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:56:17,649 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:17,649 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:56:17,651 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:17,651 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:56:17,651 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:56:17,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:17,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:17,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:17,651 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:56:17,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:17,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:17,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:17,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:17,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:17,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:17,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:17,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:56:17,654 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:56:17,654 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:56:17,654 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:56:17,654 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:56:17,654 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:56:17,654 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:56:17,655 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:17,655 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:56:17,655 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:17,655 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:17,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:17,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:17,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:17,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:17,656 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:56:17,656 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:17,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:56:17,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:17,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:17,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:17,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:17,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:17,657 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:17,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:17,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:17,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:17,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:17,659 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:56:17,659 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:17,659 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:56:17,659 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:56:17,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:56:17,659 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:56:17,659 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:17,660 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:17,660 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:17,660 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:17,660 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:17,660 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:17,660 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:56:17,659 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:56:17,660 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:56:17,660 DEBUG [Thread-3656] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:56:17,660 DEBUG [Thread-3656] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:56:17,660 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:56:17,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:56:17,742 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=360 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/.tmp/C/fcfafbe2b3bd4ab782b28b2120bc2015 2023-04-25 02:56:17,746 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/.tmp/A/1878815b560446ac8dda696440dbddfb as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/A/1878815b560446ac8dda696440dbddfb 2023-04-25 02:56:17,751 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/A/1878815b560446ac8dda696440dbddfb, entries=150, sequenceid=360, filesize=25.4 K 2023-04-25 02:56:17,752 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/.tmp/B/ac5006cce1e9465db77996c879df7734 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/B/ac5006cce1e9465db77996c879df7734 2023-04-25 02:56:17,756 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/B/ac5006cce1e9465db77996c879df7734, entries=150, sequenceid=360, filesize=11.9 K 2023-04-25 02:56:17,756 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/.tmp/C/fcfafbe2b3bd4ab782b28b2120bc2015 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/C/fcfafbe2b3bd4ab782b28b2120bc2015 2023-04-25 02:56:17,759 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/C/fcfafbe2b3bd4ab782b28b2120bc2015, entries=150, sequenceid=360, filesize=11.9 K 2023-04-25 02:56:17,760 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for d6e38fead90b2a328d380bf5577ff0ce in 877ms, sequenceid=360, compaction requested=true 2023-04-25 02:56:17,760 DEBUG [Thread-3656] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:56:17,760 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for d6e38fead90b2a328d380bf5577ff0ce: 2023-04-25 02:56:17,760 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 02:56:17,761 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-25 02:56:17,761 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-25 02:56:17,761 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 02:56:17,761 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 02:56:17,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:56:17,762 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 97750 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-25 02:56:17,762 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HStore(1912): d6e38fead90b2a328d380bf5577ff0ce/A is initiating minor compaction (all files) 2023-04-25 02:56:17,762 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of d6e38fead90b2a328d380bf5577ff0ce/A in TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. 2023-04-25 02:56:17,763 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:56:17,763 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/A/6954056042cb40a78c6c5f586eb576a7, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/A/56a6c8b8d04444d1b77062034f787a62, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/A/2c4f81230cd1496899f70b9fcfd2d61f, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/A/1878815b560446ac8dda696440dbddfb] into tmpdir=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/.tmp, totalSize=95.5 K 2023-04-25 02:56:17,763 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting 6954056042cb40a78c6c5f586eb576a7, keycount=150, bloomtype=ROW, size=26.0 K, encoding=NONE, compression=NONE, seqNum=299, earliestPutTs=1682391375046 2023-04-25 02:56:17,764 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 46914 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-25 02:56:17,764 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting 56a6c8b8d04444d1b77062034f787a62, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=336, earliestPutTs=1682391375104 2023-04-25 02:56:17,764 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HStore(1912): d6e38fead90b2a328d380bf5577ff0ce/B is initiating minor compaction (all files) 2023-04-25 02:56:17,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:56:17,764 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d6e38fead90b2a328d380bf5577ff0ce/B in TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. 2023-04-25 02:56:17,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:56:17,764 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/B/0cc319688f0c49749f96a064e514af7f, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/B/e5632252158d4d1987c1fc9973c5a9a8, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/B/2fe830a5f0ca46d097b6ad9d1c9e7380, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/B/ac5006cce1e9465db77996c879df7734] into tmpdir=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/.tmp, totalSize=45.8 K 2023-04-25 02:56:17,764 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting 2c4f81230cd1496899f70b9fcfd2d61f, keycount=100, bloomtype=ROW, size=18.6 K, encoding=NONE, compression=NONE, seqNum=349, earliestPutTs=1682391375715 2023-04-25 02:56:17,764 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:56:17,764 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:56:17,764 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:56:17,764 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:56:17,764 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting 0cc319688f0c49749f96a064e514af7f, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=299, earliestPutTs=1682391375046 2023-04-25 02:56:17,764 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting 1878815b560446ac8dda696440dbddfb, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=360, earliestPutTs=1682391376879 2023-04-25 02:56:17,765 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:17,765 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:17,765 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting e5632252158d4d1987c1fc9973c5a9a8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=336, earliestPutTs=1682391375104 2023-04-25 02:56:17,765 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting 2fe830a5f0ca46d097b6ad9d1c9e7380, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=349, earliestPutTs=1682391375715 2023-04-25 02:56:17,765 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting ac5006cce1e9465db77996c879df7734, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=360, earliestPutTs=1682391376879 2023-04-25 02:56:17,766 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:56:17,766 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:56:17,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:56:17,766 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:17,766 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:17,767 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:56:17,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:17,767 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:17,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:56:17,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:17,767 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:56:17,768 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:56:17,768 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:56:17,768 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:56:17,768 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:56:17,768 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:56:17,768 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:56:17,768 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:56:17,768 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. 2023-04-25 02:56:17,768 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. started... 2023-04-25 02:56:17,768 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d6e38fead90b2a328d380bf5577ff0ce 3/3 column families, dataSize=134.18 KB heapSize=379.45 KB 2023-04-25 02:56:17,769 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6e38fead90b2a328d380bf5577ff0ce, store=A 2023-04-25 02:56:17,769 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 02:56:17,769 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6e38fead90b2a328d380bf5577ff0ce, store=B 2023-04-25 02:56:17,769 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 02:56:17,769 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6e38fead90b2a328d380bf5577ff0ce, store=C 2023-04-25 02:56:17,769 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 02:56:17,781 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] throttle.PressureAwareThroughputController(145): d6e38fead90b2a328d380bf5577ff0ce#B#compaction#550 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 53.85 MB/second 2023-04-25 02:56:17,786 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] throttle.PressureAwareThroughputController(145): d6e38fead90b2a328d380bf5577ff0ce#A#compaction#549 average throughput is 1.41 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-25 02:56:17,805 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230425791fb55edc344a61bdd9f0f4a4085a7f to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425791fb55edc344a61bdd9f0f4a4085a7f 2023-04-25 02:56:17,807 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/.tmp/B/a2f2990f09374bf791fa5d14b7d0289b as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/B/a2f2990f09374bf791fa5d14b7d0289b 2023-04-25 02:56:17,811 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in d6e38fead90b2a328d380bf5577ff0ce/B of d6e38fead90b2a328d380bf5577ff0ce into a2f2990f09374bf791fa5d14b7d0289b(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 02:56:17,811 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d6e38fead90b2a328d380bf5577ff0ce: 2023-04-25 02:56:17,811 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce., storeName=d6e38fead90b2a328d380bf5577ff0ce/B, priority=12, startTime=1682391377761; duration=0sec 2023-04-25 02:56:17,811 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 02:56:17,811 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-25 02:56:17,812 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 46914 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-25 02:56:17,812 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HStore(1912): d6e38fead90b2a328d380bf5577ff0ce/C is initiating minor compaction (all files) 2023-04-25 02:56:17,812 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d6e38fead90b2a328d380bf5577ff0ce/C in TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. 2023-04-25 02:56:17,812 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/C/348fde98b0f1414e90d760f01955a23a, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/C/40899778f8fc4d7da81c2205efe78c6b, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/C/29c6956aa568405a8b37a6c997cfe5fb, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/C/fcfafbe2b3bd4ab782b28b2120bc2015] into tmpdir=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/.tmp, totalSize=45.8 K 2023-04-25 02:56:17,813 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting 348fde98b0f1414e90d760f01955a23a, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=299, earliestPutTs=1682391375046 2023-04-25 02:56:17,816 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting 40899778f8fc4d7da81c2205efe78c6b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=336, earliestPutTs=1682391375104 2023-04-25 02:56:17,822 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting 29c6956aa568405a8b37a6c997cfe5fb, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=349, earliestPutTs=1682391375715 2023-04-25 02:56:17,822 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting fcfafbe2b3bd4ab782b28b2120bc2015, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=360, earliestPutTs=1682391376879 2023-04-25 02:56:17,831 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] throttle.PressureAwareThroughputController(145): d6e38fead90b2a328d380bf5577ff0ce#C#compaction#552 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-25 02:56:17,841 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/.tmp/C/4c2d882cd65f4528a99e07514b30718e as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/C/4c2d882cd65f4528a99e07514b30718e 2023-04-25 02:56:17,845 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in d6e38fead90b2a328d380bf5577ff0ce/C of d6e38fead90b2a328d380bf5577ff0ce into 4c2d882cd65f4528a99e07514b30718e(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 02:56:17,845 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d6e38fead90b2a328d380bf5577ff0ce: 2023-04-25 02:56:17,845 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce., storeName=d6e38fead90b2a328d380bf5577ff0ce/C, priority=12, startTime=1682391377761; duration=0sec 2023-04-25 02:56:17,845 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 02:56:18,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(9158): Flush requested on d6e38fead90b2a328d380bf5577ff0ce 2023-04-25 02:56:18,128 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. as already flushing 2023-04-25 02:56:18,141 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:18,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39458 deadline: 1682391438139, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:18,145 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:18,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39526 deadline: 1682391438142, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:18,223 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=413, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/.tmp/A/a5710dc2cdec4c5d8f7a2ee03bd56e89 2023-04-25 02:56:18,244 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:18,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39458 deadline: 1682391438242, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:18,248 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:18,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39526 deadline: 1682391438247, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:18,447 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:18,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39458 deadline: 1682391438445, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:18,451 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:18,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39526 deadline: 1682391438450, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:18,634 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=413 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/.tmp/B/e563b5cb48194e3bb7eacda135e39c82 2023-04-25 02:56:18,749 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:18,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39458 deadline: 1682391438748, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:18,756 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:18,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39526 deadline: 1682391438754, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:19,029 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/.tmp/A/c76c65d5f3b34226ba011b0a078e4762 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/A/c76c65d5f3b34226ba011b0a078e4762 2023-04-25 02:56:19,033 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in d6e38fead90b2a328d380bf5577ff0ce/A of d6e38fead90b2a328d380bf5577ff0ce into c76c65d5f3b34226ba011b0a078e4762(size=26.2 K), total size for store is 26.2 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-04-25 02:56:19,033 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for d6e38fead90b2a328d380bf5577ff0ce: 2023-04-25 02:56:19,033 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce., storeName=d6e38fead90b2a328d380bf5577ff0ce/A, priority=12, startTime=1682391377760; duration=1sec 2023-04-25 02:56:19,034 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 02:56:19,050 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=413 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/.tmp/C/658553df4605456c87f70eeeb707f57b 2023-04-25 02:56:19,053 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/.tmp/A/a5710dc2cdec4c5d8f7a2ee03bd56e89 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/A/a5710dc2cdec4c5d8f7a2ee03bd56e89 2023-04-25 02:56:19,057 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/A/a5710dc2cdec4c5d8f7a2ee03bd56e89, entries=150, sequenceid=413, filesize=25.4 K 2023-04-25 02:56:19,057 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/.tmp/B/e563b5cb48194e3bb7eacda135e39c82 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/B/e563b5cb48194e3bb7eacda135e39c82 2023-04-25 02:56:19,060 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/B/e563b5cb48194e3bb7eacda135e39c82, entries=150, sequenceid=413, filesize=11.9 K 2023-04-25 02:56:19,061 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/.tmp/C/658553df4605456c87f70eeeb707f57b as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/C/658553df4605456c87f70eeeb707f57b 2023-04-25 02:56:19,064 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/C/658553df4605456c87f70eeeb707f57b, entries=150, sequenceid=413, filesize=11.9 K 2023-04-25 02:56:19,065 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~379.41 KB/388512, currentSize=46.96 KB/48090 for d6e38fead90b2a328d380bf5577ff0ce in 1297ms, sequenceid=413, compaction requested=false 2023-04-25 02:56:19,065 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d6e38fead90b2a328d380bf5577ff0ce: 2023-04-25 02:56:19,065 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. 2023-04-25 02:56:19,065 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:56:19,065 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:56:19,065 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:19,065 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:56:19,065 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:56:19,068 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:19,068 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:19,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:19,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:19,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:19,069 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:19,069 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:56:19,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:19,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:19,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:19,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:19,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:19,070 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:56:19,070 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5dbb8377[Count = 0] remaining members to acquire global barrier 2023-04-25 02:56:19,070 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:56:19,070 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:19,072 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:19,072 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:19,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:19,072 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:56:19,072 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:56:19,072 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:56:19,072 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:19,072 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:56:19,073 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:19,073 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:56:19,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:19,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:19,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:19,073 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:56:19,074 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:56:19,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:19,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:19,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:19,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:19,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:19,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:19,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:19,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:56:19,077 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:56:19,077 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:56:19,077 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:56:19,077 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:56:19,077 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:56:19,077 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:56:19,078 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:19,078 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:19,078 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:19,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:19,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:19,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:19,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:19,078 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:56:19,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:19,079 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:56:19,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:56:19,079 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:19,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:19,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:19,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:19,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:19,081 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:19,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:19,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:19,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:19,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:19,089 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:56:19,089 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:56:19,089 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:19,089 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:19,089 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:56:19,089 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:56:19,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:56:19,090 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:56:19,090 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:56:19,089 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:19,090 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:19,090 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:56:19,090 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:19,090 DEBUG [Thread-3656] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:56:19,090 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:56:19,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:56:19,090 DEBUG [Thread-3656] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:56:19,090 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:19,090 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:19,190 DEBUG [Thread-3656] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:56:19,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:56:19,192 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:56:19,194 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:56:19,194 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:56:19,194 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:56:19,194 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:56:19,194 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:56:19,194 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:56:19,194 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:19,194 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:19,196 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:19,196 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:56:19,196 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:56:19,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:56:19,196 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:19,196 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:56:19,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:19,197 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:19,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:56:19,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:19,197 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:56:19,197 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:56:19,197 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:56:19,197 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:56:19,197 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:56:19,197 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:56:19,198 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:56:19,198 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:56:19,198 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. 2023-04-25 02:56:19,198 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. started... 2023-04-25 02:56:19,198 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d6e38fead90b2a328d380bf5577ff0ce 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-25 02:56:19,198 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6e38fead90b2a328d380bf5577ff0ce, store=A 2023-04-25 02:56:19,198 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:56:19,198 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6e38fead90b2a328d380bf5577ff0ce, store=B 2023-04-25 02:56:19,198 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:56:19,199 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6e38fead90b2a328d380bf5577ff0ce, store=C 2023-04-25 02:56:19,199 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:56:19,215 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230425cf1520e64ed448f2802aba7176e93c81 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425cf1520e64ed448f2802aba7176e93c81 2023-04-25 02:56:19,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(9158): Flush requested on d6e38fead90b2a328d380bf5577ff0ce 2023-04-25 02:56:19,244 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. as already flushing 2023-04-25 02:56:19,268 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682391379258, max timestamp=1682391379266Num uniques 859; 2023-04-25 02:56:19,268 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-25 02:56:19,268 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682391379258, max timestamp=1682391379266Num uniques 859; 2023-04-25 02:56:19,268 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682391379258, max timestamp=1682391379266Num uniques 859; 2023-04-25 02:56:19,268 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-25 02:56:19,268 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-25 02:56:19,298 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:19,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39526 deadline: 1682391439294, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:19,298 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:19,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39458 deadline: 1682391439294, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:19,298 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:19,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39522 deadline: 1682391439295, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:19,298 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:19,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39536 deadline: 1682391439295, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:19,298 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:19,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39482 deadline: 1682391439296, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:19,402 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:19,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39526 deadline: 1682391439399, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:19,402 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:19,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39458 deadline: 1682391439399, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:19,403 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:19,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39522 deadline: 1682391439399, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:19,403 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:19,403 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:19,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39536 deadline: 1682391439399, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:19,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39482 deadline: 1682391439399, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:19,605 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:19,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39526 deadline: 1682391439603, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:19,605 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:19,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39458 deadline: 1682391439603, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:19,606 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:19,607 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39522 deadline: 1682391439605, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:19,607 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:19,607 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39536 deadline: 1682391439605, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:19,607 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:19,607 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39482 deadline: 1682391439605, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:19,626 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=426, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/.tmp/A/e99791ccc9c54fac9e4fd7bea154c5bc 2023-04-25 02:56:19,637 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=426 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/.tmp/B/afa2a8fd10ed49d583f85beca482f861 2023-04-25 02:56:19,908 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:19,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39526 deadline: 1682391439906, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:19,908 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:19,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39458 deadline: 1682391439906, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:19,911 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:19,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39482 deadline: 1682391439909, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:19,911 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:19,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39522 deadline: 1682391439909, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:19,911 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:19,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39536 deadline: 1682391439910, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:20,046 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=426 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/.tmp/C/fa941c0bffbe4e97984ea0410021fbbd 2023-04-25 02:56:20,050 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/.tmp/A/e99791ccc9c54fac9e4fd7bea154c5bc as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/A/e99791ccc9c54fac9e4fd7bea154c5bc 2023-04-25 02:56:20,054 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/A/e99791ccc9c54fac9e4fd7bea154c5bc, entries=150, sequenceid=426, filesize=25.4 K 2023-04-25 02:56:20,054 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/.tmp/B/afa2a8fd10ed49d583f85beca482f861 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/B/afa2a8fd10ed49d583f85beca482f861 2023-04-25 02:56:20,057 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/B/afa2a8fd10ed49d583f85beca482f861, entries=150, sequenceid=426, filesize=11.9 K 2023-04-25 02:56:20,058 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/.tmp/C/fa941c0bffbe4e97984ea0410021fbbd as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/C/fa941c0bffbe4e97984ea0410021fbbd 2023-04-25 02:56:20,061 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/C/fa941c0bffbe4e97984ea0410021fbbd, entries=150, sequenceid=426, filesize=11.9 K 2023-04-25 02:56:20,062 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for d6e38fead90b2a328d380bf5577ff0ce in 864ms, sequenceid=426, compaction requested=true 2023-04-25 02:56:20,062 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d6e38fead90b2a328d380bf5577ff0ce: 2023-04-25 02:56:20,062 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. 2023-04-25 02:56:20,062 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:56:20,062 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:56:20,062 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:20,062 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:56:20,062 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:56:20,063 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:20,064 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:20,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:20,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:20,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:20,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:20,064 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:20,064 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:56:20,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:20,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:20,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:20,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:20,065 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:56:20,065 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@615cb3a7[Count = 0] remaining members to acquire global barrier 2023-04-25 02:56:20,065 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:56:20,066 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:20,067 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:20,067 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:20,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:20,067 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:56:20,067 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:56:20,067 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:20,067 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:56:20,067 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:56:20,072 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:56:20,072 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:56:20,072 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:20,072 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:56:20,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:20,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:20,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:20,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:20,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:20,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:20,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:20,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:20,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:20,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:20,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:56:20,075 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:56:20,075 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:56:20,075 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:56:20,075 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:56:20,075 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:56:20,075 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:56:20,077 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:20,077 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:20,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:20,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:20,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:20,077 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:20,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:20,077 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:56:20,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:20,078 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:20,078 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:56:20,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:56:20,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:20,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:20,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:20,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:20,079 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:20,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:20,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:20,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:20,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:20,081 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:56:20,081 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:56:20,081 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:56:20,081 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:56:20,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:56:20,081 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:56:20,081 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:56:20,081 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:56:20,081 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:20,082 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:20,082 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:20,082 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:20,082 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:56:20,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:56:20,082 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:20,082 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:20,082 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:20,088 DEBUG [Thread-3656] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:56:20,088 DEBUG [Thread-3656] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:56:20,188 DEBUG [Thread-3656] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:56:20,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:56:20,190 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:56:20,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:56:20,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:56:20,192 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:56:20,192 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:56:20,192 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:56:20,192 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:56:20,192 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:20,192 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:20,193 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:56:20,193 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:20,193 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:56:20,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:56:20,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:20,194 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:20,194 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:56:20,194 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:20,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:56:20,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:20,195 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:56:20,195 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:56:20,195 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:56:20,195 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:56:20,195 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:56:20,195 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:56:20,195 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:56:20,195 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. 2023-04-25 02:56:20,195 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. started... 2023-04-25 02:56:20,195 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:56:20,196 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d6e38fead90b2a328d380bf5577ff0ce 3/3 column families, dataSize=134.18 KB heapSize=380.30 KB 2023-04-25 02:56:20,196 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6e38fead90b2a328d380bf5577ff0ce, store=A 2023-04-25 02:56:20,196 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 02:56:20,197 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6e38fead90b2a328d380bf5577ff0ce, store=B 2023-04-25 02:56:20,197 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 02:56:20,197 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6e38fead90b2a328d380bf5577ff0ce, store=C 2023-04-25 02:56:20,197 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 02:56:20,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(9158): Flush requested on d6e38fead90b2a328d380bf5577ff0ce 2023-04-25 02:56:20,414 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. as already flushing 2023-04-25 02:56:20,421 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:20,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39536 deadline: 1682391440416, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:20,424 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:20,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39482 deadline: 1682391440419, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:20,426 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:20,426 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:20,426 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39458 deadline: 1682391440420, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:20,426 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39526 deadline: 1682391440421, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:20,426 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:20,426 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39522 deadline: 1682391440421, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:20,524 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:20,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39536 deadline: 1682391440522, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:20,527 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:20,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39482 deadline: 1682391440525, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:20,529 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:20,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39526 deadline: 1682391440527, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:20,530 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:20,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39458 deadline: 1682391440527, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:20,530 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:20,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39522 deadline: 1682391440527, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:20,611 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230425eda77ab9ff5d4662a22eae6a5735f123 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425eda77ab9ff5d4662a22eae6a5735f123 2023-04-25 02:56:20,616 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=462, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/.tmp/A/5e743c072b45416abbc4c1041bfd2d27 2023-04-25 02:56:20,627 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=462 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/.tmp/B/69e38add9fe743eda6f8c8fe7f4edb8e 2023-04-25 02:56:20,639 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=462 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/.tmp/C/8f7df7fffc89443f9256d967bcda2ff1 2023-04-25 02:56:20,644 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/.tmp/A/5e743c072b45416abbc4c1041bfd2d27 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/A/5e743c072b45416abbc4c1041bfd2d27 2023-04-25 02:56:20,648 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/A/5e743c072b45416abbc4c1041bfd2d27, entries=150, sequenceid=462, filesize=25.4 K 2023-04-25 02:56:20,649 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/.tmp/B/69e38add9fe743eda6f8c8fe7f4edb8e as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/B/69e38add9fe743eda6f8c8fe7f4edb8e 2023-04-25 02:56:20,652 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/B/69e38add9fe743eda6f8c8fe7f4edb8e, entries=150, sequenceid=462, filesize=11.9 K 2023-04-25 02:56:20,652 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/.tmp/C/8f7df7fffc89443f9256d967bcda2ff1 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/C/8f7df7fffc89443f9256d967bcda2ff1 2023-04-25 02:56:20,656 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/C/8f7df7fffc89443f9256d967bcda2ff1, entries=150, sequenceid=462, filesize=11.9 K 2023-04-25 02:56:20,657 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~380.25 KB/389376, currentSize=53.67 KB/54960 for d6e38fead90b2a328d380bf5577ff0ce in 461ms, sequenceid=462, compaction requested=true 2023-04-25 02:56:20,657 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d6e38fead90b2a328d380bf5577ff0ce: 2023-04-25 02:56:20,657 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. 2023-04-25 02:56:20,657 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:56:20,657 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:56:20,657 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:20,657 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:56:20,657 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:56:20,658 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:20,658 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:20,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:20,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:20,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:20,659 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:20,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:20,659 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:56:20,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:20,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:20,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:20,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:20,661 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:56:20,661 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@be75c08[Count = 0] remaining members to acquire global barrier 2023-04-25 02:56:20,661 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:56:20,661 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:20,668 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:20,668 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:20,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:20,668 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:56:20,668 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:56:20,668 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:56:20,668 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:20,668 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:56:20,669 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:56:20,669 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:20,669 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:56:20,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:20,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:20,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:20,669 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:56:20,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:20,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:20,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:20,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:20,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:20,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:20,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:20,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:56:20,672 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:56:20,672 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:56:20,672 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:56:20,672 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:56:20,672 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:56:20,672 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:56:20,674 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:20,674 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:56:20,674 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:20,674 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:20,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:20,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:20,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:20,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:20,674 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:20,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:20,674 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:56:20,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:56:20,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:20,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:20,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:20,675 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:20,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:20,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:20,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:20,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:20,678 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:56:20,678 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:20,678 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:56:20,678 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:56:20,678 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:56:20,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:56:20,678 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:20,678 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:56:20,678 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:56:20,678 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:20,678 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:20,678 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:20,678 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:56:20,678 DEBUG [Thread-3656] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:56:20,678 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:56:20,678 DEBUG [Thread-3656] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:56:20,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:56:20,678 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:20,679 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:20,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(9158): Flush requested on d6e38fead90b2a328d380bf5577ff0ce 2023-04-25 02:56:20,728 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing d6e38fead90b2a328d380bf5577ff0ce 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-25 02:56:20,728 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6e38fead90b2a328d380bf5577ff0ce, store=A 2023-04-25 02:56:20,728 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:56:20,728 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6e38fead90b2a328d380bf5577ff0ce, store=B 2023-04-25 02:56:20,728 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:56:20,728 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6e38fead90b2a328d380bf5577ff0ce, store=C 2023-04-25 02:56:20,728 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:56:20,740 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023042508c4c46548354735ac1f62cd70d54b41 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042508c4c46548354735ac1f62cd70d54b41 2023-04-25 02:56:20,747 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:20,748 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39458 deadline: 1682391440743, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:20,749 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:20,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39522 deadline: 1682391440745, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:20,750 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:20,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 204 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39526 deadline: 1682391440746, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:20,751 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:20,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39536 deadline: 1682391440747, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:20,751 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:20,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39482 deadline: 1682391440748, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:20,779 DEBUG [Thread-3656] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:56:20,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:56:20,781 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:56:20,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:56:20,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:56:20,782 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:56:20,782 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:56:20,782 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:56:20,782 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:56:20,783 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:20,783 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:20,785 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:56:20,785 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:20,785 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:56:20,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:56:20,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:20,785 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:20,785 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:56:20,786 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:20,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:56:20,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:20,786 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:56:20,786 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:56:20,786 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:56:20,786 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:56:20,786 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:56:20,786 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:56:20,786 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:56:20,787 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:56:20,787 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. 2023-04-25 02:56:20,787 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. started... 2023-04-25 02:56:20,787 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. as already flushing 2023-04-25 02:56:20,787 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. 2023-04-25 02:56:20,787 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:56:20,787 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:56:20,787 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:20,787 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:56:20,787 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:56:20,788 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:20,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:20,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:20,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:20,789 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:20,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:20,789 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:20,789 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:56:20,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:20,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:20,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:20,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:20,790 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:56:20,790 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@30321641[Count = 0] remaining members to acquire global barrier 2023-04-25 02:56:20,790 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:56:20,790 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:20,791 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:20,791 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:20,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:20,791 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:56:20,791 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:20,791 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:56:20,791 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:56:20,792 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:56:20,793 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:56:20,793 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:20,793 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:56:20,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:20,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:20,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:20,793 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:56:20,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:20,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:20,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:20,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:20,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:20,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:20,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:20,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:56:20,795 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:56:20,795 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:56:20,795 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:56:20,795 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:56:20,795 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:56:20,795 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:56:20,797 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:20,798 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:56:20,798 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:20,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:20,798 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:20,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:20,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:20,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:20,798 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:56:20,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:56:20,798 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:20,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:20,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:20,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:20,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:20,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:20,800 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:20,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:20,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:20,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:20,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:20,802 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:56:20,802 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:56:20,802 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:56:20,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:56:20,802 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:56:20,802 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:56:20,802 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:56:20,802 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:56:20,802 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:20,802 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:56:20,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:56:20,802 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:20,802 DEBUG [Thread-3656] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:56:20,803 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:20,803 DEBUG [Thread-3656] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:56:20,803 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:20,803 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:20,803 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:20,803 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:20,850 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:20,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39458 deadline: 1682391440848, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:20,851 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:20,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39522 deadline: 1682391440850, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:20,853 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:20,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 206 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39526 deadline: 1682391440851, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:20,853 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:20,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39536 deadline: 1682391440851, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:20,854 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:20,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39482 deadline: 1682391440852, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:20,903 DEBUG [Thread-3656] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:56:20,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:56:20,905 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:56:20,906 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:56:20,906 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:56:20,906 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:56:20,906 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:56:20,906 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:56:20,906 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:56:20,907 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:20,907 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:20,908 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:20,908 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:56:20,908 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:56:20,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:56:20,908 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:20,908 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:56:20,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:20,909 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:20,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:56:20,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:20,909 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:56:20,909 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:56:20,909 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:56:20,909 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:56:20,909 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:56:20,910 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:56:20,910 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:56:20,910 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:56:20,910 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. 2023-04-25 02:56:20,910 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. started... 2023-04-25 02:56:20,910 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. as already flushing 2023-04-25 02:56:20,910 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. 2023-04-25 02:56:20,910 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:56:20,910 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:56:20,910 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:20,910 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:56:20,910 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:56:20,911 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:20,911 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:20,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:20,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:20,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:20,911 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:20,912 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:56:20,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:20,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:20,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:20,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:20,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:20,913 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:56:20,913 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@31939b0b[Count = 0] remaining members to acquire global barrier 2023-04-25 02:56:20,913 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:56:20,913 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:20,915 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:20,915 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:20,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:20,915 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:56:20,915 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:56:20,915 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:56:20,915 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:20,915 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:56:20,916 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:56:20,916 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:56:20,916 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:20,916 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:56:20,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:20,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:20,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:20,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:20,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:20,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:20,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:20,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:20,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:20,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:20,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:56:20,918 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:56:20,918 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:56:20,918 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:56:20,918 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:56:20,918 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:56:20,918 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:56:20,920 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:20,920 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:56:20,920 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:20,920 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:20,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:20,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:20,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:20,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:20,920 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:56:20,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:20,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:56:20,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:20,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:20,921 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:20,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:20,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:20,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:20,923 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:20,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:20,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:20,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:20,928 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:56:20,928 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:20,928 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:56:20,928 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:56:20,928 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:20,928 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:56:20,928 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:20,928 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:56:20,928 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:56:20,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:56:20,928 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:20,928 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:20,928 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:20,928 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:20,928 DEBUG [Thread-3656] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:56:20,929 DEBUG [Thread-3656] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:56:20,929 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:56:20,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:56:21,029 DEBUG [Thread-3656] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:56:21,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:56:21,030 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:56:21,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:56:21,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:56:21,032 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:56:21,032 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:56:21,032 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:56:21,032 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:56:21,032 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:21,032 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:21,034 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:21,034 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:56:21,034 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:56:21,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:56:21,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:21,034 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:21,034 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:56:21,035 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:21,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:56:21,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:21,035 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:56:21,035 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:56:21,035 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:56:21,035 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:56:21,035 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:56:21,035 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:56:21,036 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:56:21,036 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:56:21,036 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. 2023-04-25 02:56:21,036 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. started... 2023-04-25 02:56:21,036 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. as already flushing 2023-04-25 02:56:21,036 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. 2023-04-25 02:56:21,036 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:56:21,036 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:56:21,036 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:21,036 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:56:21,036 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:56:21,037 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:21,037 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:21,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:21,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:21,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:21,038 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:21,038 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:56:21,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:21,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:21,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:21,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:21,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:21,039 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:56:21,039 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3556c715[Count = 0] remaining members to acquire global barrier 2023-04-25 02:56:21,039 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:56:21,039 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:21,040 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:21,040 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:21,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:21,040 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:56:21,041 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:56:21,041 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:56:21,041 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:21,041 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:56:21,043 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:56:21,043 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:21,043 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:56:21,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:21,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:21,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:21,043 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:56:21,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:21,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:21,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:21,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:21,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:21,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:21,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:21,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:56:21,045 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:56:21,045 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:56:21,045 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:56:21,045 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:56:21,045 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:56:21,045 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:56:21,047 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:21,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:21,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:21,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:21,047 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:21,047 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:56:21,047 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:21,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:21,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:21,047 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:56:21,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:56:21,048 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:21,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:21,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:21,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:21,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:21,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:21,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:21,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:21,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:21,050 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:21,052 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:56:21,052 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:56:21,052 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:56:21,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:56:21,052 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:21,052 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:21,052 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:21,052 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:56:21,052 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:56:21,052 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:56:21,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:56:21,052 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:56:21,052 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:21,052 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:21,052 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:21,052 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:21,052 DEBUG [Thread-3656] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:56:21,053 DEBUG [Thread-3656] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:56:21,055 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:21,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39458 deadline: 1682391441052, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:21,055 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:21,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39522 deadline: 1682391441053, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:21,058 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:21,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 208 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39526 deadline: 1682391441054, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:21,058 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:21,058 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:21,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39482 deadline: 1682391441054, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:21,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39536 deadline: 1682391441055, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:21,148 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=474, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/.tmp/A/71799a1a2011498b9677e893d90f84fb 2023-04-25 02:56:21,153 DEBUG [Thread-3656] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:56:21,153 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:56:21,155 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:56:21,158 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=474 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/.tmp/B/86d489ab619049beb472de16eb552058 2023-04-25 02:56:21,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:56:21,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:56:21,160 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:56:21,160 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:56:21,160 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:56:21,160 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:56:21,161 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:21,161 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:21,162 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:56:21,162 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:21,162 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:56:21,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:56:21,163 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:21,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:21,163 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:56:21,163 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:21,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:56:21,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:21,164 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:56:21,164 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:56:21,164 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:56:21,164 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:56:21,164 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:56:21,164 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:56:21,164 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:56:21,165 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:56:21,165 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. 2023-04-25 02:56:21,165 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. started... 2023-04-25 02:56:21,165 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. as already flushing 2023-04-25 02:56:21,165 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. 2023-04-25 02:56:21,165 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:56:21,165 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:56:21,165 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:21,165 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:56:21,165 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:56:21,168 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:21,168 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:21,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:21,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:21,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:21,168 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:21,168 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:56:21,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:21,169 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=474 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/.tmp/C/79fa302c7857479e8b8694a3dde2806d 2023-04-25 02:56:21,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:21,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:21,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:21,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:21,171 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:56:21,171 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@460caab[Count = 0] remaining members to acquire global barrier 2023-04-25 02:56:21,171 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:56:21,171 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:21,172 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:21,172 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:21,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:21,172 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:56:21,172 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:56:21,172 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:56:21,172 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:21,172 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:56:21,173 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/.tmp/A/71799a1a2011498b9677e893d90f84fb as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/A/71799a1a2011498b9677e893d90f84fb 2023-04-25 02:56:21,173 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:21,173 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:56:21,174 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:56:21,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:21,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:21,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:21,174 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:56:21,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:21,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:21,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:21,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:21,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:21,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:21,176 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/A/71799a1a2011498b9677e893d90f84fb, entries=200, sequenceid=474, filesize=32.3 K 2023-04-25 02:56:21,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:21,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:56:21,177 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:56:21,177 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:56:21,177 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:56:21,177 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:56:21,177 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:56:21,177 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:56:21,177 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/.tmp/B/86d489ab619049beb472de16eb552058 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/B/86d489ab619049beb472de16eb552058 2023-04-25 02:56:21,178 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:21,178 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:56:21,178 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:21,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:21,178 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:21,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:21,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:21,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:21,179 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:56:21,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:56:21,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:21,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:21,179 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:21,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:21,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:21,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:21,181 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:21,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:21,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:21,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:21,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:21,183 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:21,183 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:56:21,183 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:21,183 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:56:21,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:56:21,183 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:56:21,183 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:56:21,183 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:21,184 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:21,184 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:21,184 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:21,184 DEBUG [Thread-3656] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:56:21,183 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:56:21,184 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:56:21,183 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:56:21,184 DEBUG [Thread-3656] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:56:21,184 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:21,184 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:56:21,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:56:21,185 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/B/86d489ab619049beb472de16eb552058, entries=150, sequenceid=474, filesize=11.9 K 2023-04-25 02:56:21,186 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/.tmp/C/79fa302c7857479e8b8694a3dde2806d as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/C/79fa302c7857479e8b8694a3dde2806d 2023-04-25 02:56:21,189 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/C/79fa302c7857479e8b8694a3dde2806d, entries=150, sequenceid=474, filesize=11.9 K 2023-04-25 02:56:21,189 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for d6e38fead90b2a328d380bf5577ff0ce in 461ms, sequenceid=474, compaction requested=true 2023-04-25 02:56:21,190 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for d6e38fead90b2a328d380bf5577ff0ce: 2023-04-25 02:56:21,190 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 02:56:21,190 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-25 02:56:21,190 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-25 02:56:21,190 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 02:56:21,190 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 02:56:21,191 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 138020 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-25 02:56:21,191 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HStore(1912): d6e38fead90b2a328d380bf5577ff0ce/A is initiating minor compaction (all files) 2023-04-25 02:56:21,191 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of d6e38fead90b2a328d380bf5577ff0ce/A in TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. 2023-04-25 02:56:21,191 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/A/c76c65d5f3b34226ba011b0a078e4762, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/A/a5710dc2cdec4c5d8f7a2ee03bd56e89, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/A/e99791ccc9c54fac9e4fd7bea154c5bc, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/A/5e743c072b45416abbc4c1041bfd2d27, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/A/71799a1a2011498b9677e893d90f84fb] into tmpdir=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/.tmp, totalSize=134.8 K 2023-04-25 02:56:21,192 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61680 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-25 02:56:21,192 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HStore(1912): d6e38fead90b2a328d380bf5577ff0ce/B is initiating minor compaction (all files) 2023-04-25 02:56:21,192 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d6e38fead90b2a328d380bf5577ff0ce/B in TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. 2023-04-25 02:56:21,192 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/B/a2f2990f09374bf791fa5d14b7d0289b, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/B/e563b5cb48194e3bb7eacda135e39c82, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/B/afa2a8fd10ed49d583f85beca482f861, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/B/69e38add9fe743eda6f8c8fe7f4edb8e, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/B/86d489ab619049beb472de16eb552058] into tmpdir=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/.tmp, totalSize=60.2 K 2023-04-25 02:56:21,192 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting c76c65d5f3b34226ba011b0a078e4762, keycount=150, bloomtype=ROW, size=26.2 K, encoding=NONE, compression=NONE, seqNum=360, earliestPutTs=1682391376879 2023-04-25 02:56:21,192 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting a2f2990f09374bf791fa5d14b7d0289b, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=360, earliestPutTs=1682391376879 2023-04-25 02:56:21,192 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting a5710dc2cdec4c5d8f7a2ee03bd56e89, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=413, earliestPutTs=1682391377000 2023-04-25 02:56:21,193 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting e563b5cb48194e3bb7eacda135e39c82, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=413, earliestPutTs=1682391377000 2023-04-25 02:56:21,193 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting e99791ccc9c54fac9e4fd7bea154c5bc, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=426, earliestPutTs=1682391378135 2023-04-25 02:56:21,193 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting afa2a8fd10ed49d583f85beca482f861, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=426, earliestPutTs=1682391378135 2023-04-25 02:56:21,193 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting 5e743c072b45416abbc4c1041bfd2d27, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=462, earliestPutTs=1682391379293 2023-04-25 02:56:21,193 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting 69e38add9fe743eda6f8c8fe7f4edb8e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=462, earliestPutTs=1682391379293 2023-04-25 02:56:21,193 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting 71799a1a2011498b9677e893d90f84fb, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=474, earliestPutTs=1682391380413 2023-04-25 02:56:21,194 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting 86d489ab619049beb472de16eb552058, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=474, earliestPutTs=1682391380420 2023-04-25 02:56:21,203 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] throttle.PressureAwareThroughputController(145): d6e38fead90b2a328d380bf5577ff0ce#B#compaction#565 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 53.85 MB/second 2023-04-25 02:56:21,213 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] throttle.PressureAwareThroughputController(145): d6e38fead90b2a328d380bf5577ff0ce#A#compaction#564 average throughput is 1.64 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-25 02:56:21,225 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/.tmp/B/ed0b2eab0f3f4e6aa80995c7cfd3af38 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/B/ed0b2eab0f3f4e6aa80995c7cfd3af38 2023-04-25 02:56:21,230 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in d6e38fead90b2a328d380bf5577ff0ce/B of d6e38fead90b2a328d380bf5577ff0ce into ed0b2eab0f3f4e6aa80995c7cfd3af38(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 02:56:21,230 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d6e38fead90b2a328d380bf5577ff0ce: 2023-04-25 02:56:21,230 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce., storeName=d6e38fead90b2a328d380bf5577ff0ce/B, priority=11, startTime=1682391381190; duration=0sec 2023-04-25 02:56:21,231 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 02:56:21,231 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-25 02:56:21,233 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61680 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-25 02:56:21,233 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HStore(1912): d6e38fead90b2a328d380bf5577ff0ce/C is initiating minor compaction (all files) 2023-04-25 02:56:21,233 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d6e38fead90b2a328d380bf5577ff0ce/C in TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. 2023-04-25 02:56:21,233 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/C/4c2d882cd65f4528a99e07514b30718e, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/C/658553df4605456c87f70eeeb707f57b, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/C/fa941c0bffbe4e97984ea0410021fbbd, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/C/8f7df7fffc89443f9256d967bcda2ff1, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/C/79fa302c7857479e8b8694a3dde2806d] into tmpdir=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/.tmp, totalSize=60.2 K 2023-04-25 02:56:21,233 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting 4c2d882cd65f4528a99e07514b30718e, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=360, earliestPutTs=1682391376879 2023-04-25 02:56:21,234 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting 658553df4605456c87f70eeeb707f57b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=413, earliestPutTs=1682391377000 2023-04-25 02:56:21,234 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting fa941c0bffbe4e97984ea0410021fbbd, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=426, earliestPutTs=1682391378135 2023-04-25 02:56:21,235 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting 8f7df7fffc89443f9256d967bcda2ff1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=462, earliestPutTs=1682391379293 2023-04-25 02:56:21,235 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/.tmp/A/dc713e1bf01845d2bb5c555f981ea075 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/A/dc713e1bf01845d2bb5c555f981ea075 2023-04-25 02:56:21,235 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting 79fa302c7857479e8b8694a3dde2806d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=474, earliestPutTs=1682391380420 2023-04-25 02:56:21,239 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in d6e38fead90b2a328d380bf5577ff0ce/A of d6e38fead90b2a328d380bf5577ff0ce into dc713e1bf01845d2bb5c555f981ea075(size=26.3 K), total size for store is 26.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 02:56:21,239 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for d6e38fead90b2a328d380bf5577ff0ce: 2023-04-25 02:56:21,239 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce., storeName=d6e38fead90b2a328d380bf5577ff0ce/A, priority=11, startTime=1682391381190; duration=0sec 2023-04-25 02:56:21,239 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 02:56:21,248 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] throttle.PressureAwareThroughputController(145): d6e38fead90b2a328d380bf5577ff0ce#C#compaction#566 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-25 02:56:21,284 DEBUG [Thread-3656] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:56:21,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:56:21,286 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:56:21,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:56:21,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:56:21,288 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:56:21,289 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:56:21,289 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:56:21,289 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:56:21,289 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:21,289 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:21,290 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:56:21,290 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:21,290 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:56:21,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:56:21,291 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:21,291 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:56:21,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:21,291 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:21,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:56:21,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:21,291 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:56:21,292 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:56:21,292 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:56:21,292 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:56:21,292 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:56:21,292 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:56:21,292 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:56:21,292 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:56:21,292 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. 2023-04-25 02:56:21,292 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. started... 2023-04-25 02:56:21,292 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d6e38fead90b2a328d380bf5577ff0ce 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-25 02:56:21,292 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6e38fead90b2a328d380bf5577ff0ce, store=A 2023-04-25 02:56:21,293 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:56:21,293 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6e38fead90b2a328d380bf5577ff0ce, store=B 2023-04-25 02:56:21,293 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:56:21,293 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6e38fead90b2a328d380bf5577ff0ce, store=C 2023-04-25 02:56:21,293 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:56:21,313 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304259be9a2ecc4bf48bd91e410b991fd7634 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304259be9a2ecc4bf48bd91e410b991fd7634 2023-04-25 02:56:21,318 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=496, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/.tmp/A/40becd105c724ef18f63221c2eed5e72 2023-04-25 02:56:21,327 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=496 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/.tmp/B/7f453a30c1f7459ebce14d9efa60f2f3 2023-04-25 02:56:21,345 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=496 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/.tmp/C/af3e692632b44faba1a53b6f3a6331e3 2023-04-25 02:56:21,352 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/.tmp/A/40becd105c724ef18f63221c2eed5e72 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/A/40becd105c724ef18f63221c2eed5e72 2023-04-25 02:56:21,356 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/A/40becd105c724ef18f63221c2eed5e72, entries=150, sequenceid=496, filesize=25.4 K 2023-04-25 02:56:21,357 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/.tmp/B/7f453a30c1f7459ebce14d9efa60f2f3 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/B/7f453a30c1f7459ebce14d9efa60f2f3 2023-04-25 02:56:21,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(9158): Flush requested on d6e38fead90b2a328d380bf5577ff0ce 2023-04-25 02:56:21,358 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. as already flushing 2023-04-25 02:56:21,361 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/B/7f453a30c1f7459ebce14d9efa60f2f3, entries=150, sequenceid=496, filesize=11.9 K 2023-04-25 02:56:21,361 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/.tmp/C/af3e692632b44faba1a53b6f3a6331e3 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/C/af3e692632b44faba1a53b6f3a6331e3 2023-04-25 02:56:21,365 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/C/af3e692632b44faba1a53b6f3a6331e3, entries=150, sequenceid=496, filesize=11.9 K 2023-04-25 02:56:21,366 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=6.71 KB/6870 for d6e38fead90b2a328d380bf5577ff0ce in 74ms, sequenceid=496, compaction requested=false 2023-04-25 02:56:21,366 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d6e38fead90b2a328d380bf5577ff0ce: 2023-04-25 02:56:21,366 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. 2023-04-25 02:56:21,366 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:56:21,366 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:56:21,366 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:21,366 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:56:21,366 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:56:21,368 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:21,368 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:21,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:21,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:21,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:21,368 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:21,368 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:56:21,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:21,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:21,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:21,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:21,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:21,370 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:56:21,370 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@24c04478[Count = 0] remaining members to acquire global barrier 2023-04-25 02:56:21,370 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:56:21,370 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:21,372 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:21,372 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:21,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:21,372 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:56:21,372 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:56:21,372 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:56:21,372 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:21,372 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:56:21,374 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:21,374 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:56:21,374 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:56:21,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:21,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:21,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:21,374 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:56:21,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:21,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:21,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:21,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:21,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:21,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:21,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:21,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:56:21,377 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:56:21,377 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:56:21,377 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:56:21,377 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:56:21,377 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:56:21,377 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:56:21,379 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:21,379 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:21,379 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:21,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:21,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:21,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:21,379 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:56:21,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:21,379 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:56:21,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:21,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:56:21,379 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:21,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:21,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:21,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:21,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:21,381 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:21,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:21,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:21,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:21,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:21,386 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(9158): Flush requested on d6e38fead90b2a328d380bf5577ff0ce 2023-04-25 02:56:21,386 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:21,386 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:21,386 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:21,386 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing d6e38fead90b2a328d380bf5577ff0ce 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-25 02:56:21,387 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:21,387 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:21,387 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:21,387 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:21,387 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:56:21,387 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:56:21,387 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:56:21,387 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:56:21,387 DEBUG [Thread-3656] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:56:21,387 DEBUG [Thread-3656] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:56:21,392 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:56:21,392 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:56:21,392 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:56:21,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:56:21,392 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:56:21,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:56:21,393 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6e38fead90b2a328d380bf5577ff0ce, store=A 2023-04-25 02:56:21,393 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:56:21,393 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6e38fead90b2a328d380bf5577ff0ce, store=B 2023-04-25 02:56:21,393 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:56:21,393 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6e38fead90b2a328d380bf5577ff0ce, store=C 2023-04-25 02:56:21,393 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:56:21,411 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:21,411 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:21,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39522 deadline: 1682391441406, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:21,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 215 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39526 deadline: 1682391441407, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:21,411 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:21,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39482 deadline: 1682391441409, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:21,414 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:21,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 208 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39458 deadline: 1682391441410, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:21,414 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:21,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39536 deadline: 1682391441411, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:21,487 DEBUG [Thread-3656] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:56:21,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:56:21,489 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:56:21,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:56:21,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:56:21,490 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:56:21,491 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:56:21,491 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:56:21,491 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:56:21,491 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:21,491 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:21,493 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:56:21,493 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:21,493 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:56:21,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:56:21,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:21,494 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:21,494 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:56:21,494 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:21,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:56:21,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:21,494 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:56:21,495 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:56:21,495 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:56:21,495 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:56:21,495 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:56:21,495 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:56:21,495 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:56:21,495 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:56:21,495 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. 2023-04-25 02:56:21,495 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. started... 2023-04-25 02:56:21,495 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. as already flushing 2023-04-25 02:56:21,495 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. 2023-04-25 02:56:21,496 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:56:21,496 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:56:21,496 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:21,496 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:56:21,496 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:56:21,497 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:21,497 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:21,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:21,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:21,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:21,497 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:21,497 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:56:21,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:21,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:21,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:21,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:21,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:21,499 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:56:21,499 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@9a6d828[Count = 0] remaining members to acquire global barrier 2023-04-25 02:56:21,499 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:56:21,499 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:21,500 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:21,500 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:21,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:21,500 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:56:21,500 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:56:21,500 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:56:21,500 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:21,500 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:56:21,501 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:56:21,501 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:56:21,501 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:21,501 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:56:21,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:21,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:21,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:21,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:21,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:21,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:21,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:21,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:21,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:21,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:21,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:56:21,504 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:56:21,504 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:56:21,504 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:56:21,504 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:56:21,504 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:56:21,504 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:56:21,506 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:21,506 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:21,506 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:21,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:21,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:21,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:21,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:21,506 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:56:21,507 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:56:21,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:56:21,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:21,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:21,508 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:21,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:21,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:21,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:21,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:21,509 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:21,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:21,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:21,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:21,511 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:56:21,511 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:56:21,511 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:21,511 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:56:21,511 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:56:21,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:56:21,511 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:56:21,511 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:56:21,511 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:56:21,511 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:21,511 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:21,511 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:21,511 DEBUG [Thread-3656] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:56:21,511 DEBUG [Thread-3656] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:56:21,511 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:56:21,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:56:21,511 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:21,512 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:21,512 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:21,513 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:21,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 217 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39526 deadline: 1682391441513, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:21,514 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:21,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39482 deadline: 1682391441513, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:21,514 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:21,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39522 deadline: 1682391441513, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:21,516 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:21,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39458 deadline: 1682391441515, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:21,516 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:21,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39536 deadline: 1682391441515, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:21,611 DEBUG [Thread-3656] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:56:21,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:56:21,614 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:56:21,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:56:21,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:56:21,615 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:56:21,615 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:56:21,615 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:56:21,615 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:56:21,615 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:21,615 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:21,616 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:21,617 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:56:21,617 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:56:21,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:56:21,617 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:21,617 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:56:21,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:21,618 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:21,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:56:21,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:21,618 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:56:21,618 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:56:21,618 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:56:21,619 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:56:21,619 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:56:21,619 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:56:21,619 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:56:21,619 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:56:21,619 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. 2023-04-25 02:56:21,619 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. started... 2023-04-25 02:56:21,619 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. as already flushing 2023-04-25 02:56:21,619 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. 2023-04-25 02:56:21,619 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:56:21,619 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:56:21,619 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:21,619 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:56:21,619 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:56:21,621 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:21,621 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:21,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:21,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:21,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:21,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:21,621 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:21,621 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:56:21,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:21,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:21,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:21,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:21,623 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:56:21,623 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@acbc866[Count = 0] remaining members to acquire global barrier 2023-04-25 02:56:21,623 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:56:21,623 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:21,625 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:21,625 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:21,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:21,625 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:56:21,625 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:21,625 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:56:21,625 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:56:21,625 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:56:21,627 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:21,627 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:56:21,627 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:56:21,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:21,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:21,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:21,627 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:56:21,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:21,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:21,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:21,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:21,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:21,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:21,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:21,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:56:21,629 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:56:21,629 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:56:21,629 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:56:21,629 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:56:21,629 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:56:21,629 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:56:21,631 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:21,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:21,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:21,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:21,631 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:21,631 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:56:21,631 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:21,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:21,632 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:21,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:21,632 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:56:21,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:56:21,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:21,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:21,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:21,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:21,633 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:21,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:21,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:21,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:21,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:21,636 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:56:21,636 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:56:21,636 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:56:21,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:56:21,636 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:56:21,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:56:21,636 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:21,636 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:21,636 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:21,636 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:56:21,636 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:56:21,637 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:56:21,636 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:21,637 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:21,637 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:21,637 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:21,637 DEBUG [Thread-3656] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:56:21,637 DEBUG [Thread-3656] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:56:21,637 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:56:21,666 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/.tmp/C/bfb26d8a29014cffada1706dc430ead0 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/C/bfb26d8a29014cffada1706dc430ead0 2023-04-25 02:56:21,670 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in d6e38fead90b2a328d380bf5577ff0ce/C of d6e38fead90b2a328d380bf5577ff0ce into bfb26d8a29014cffada1706dc430ead0(size=12.8 K), total size for store is 24.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 02:56:21,670 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d6e38fead90b2a328d380bf5577ff0ce: 2023-04-25 02:56:21,670 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce., storeName=d6e38fead90b2a328d380bf5577ff0ce/C, priority=11, startTime=1682391381190; duration=0sec 2023-04-25 02:56:21,671 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 02:56:21,716 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:21,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 219 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39526 deadline: 1682391441714, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:21,717 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:21,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39522 deadline: 1682391441715, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:21,718 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:21,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39482 deadline: 1682391441716, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:21,719 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:21,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39458 deadline: 1682391441717, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:21,719 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:21,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39536 deadline: 1682391441717, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:21,737 DEBUG [Thread-3656] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:56:21,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:56:21,741 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:56:21,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:56:21,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:56:21,742 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:56:21,742 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:56:21,742 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:56:21,742 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:56:21,742 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:21,742 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:21,744 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:21,744 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:56:21,745 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:56:21,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:56:21,745 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:21,745 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:56:21,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:21,745 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:21,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:56:21,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:21,746 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:56:21,746 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:56:21,746 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:56:21,746 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:56:21,746 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:56:21,746 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:56:21,746 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:56:21,746 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:56:21,746 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. 2023-04-25 02:56:21,746 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. started... 2023-04-25 02:56:21,747 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. as already flushing 2023-04-25 02:56:21,747 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. 2023-04-25 02:56:21,747 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:56:21,747 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:56:21,747 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:21,747 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:56:21,747 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:56:21,749 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:21,749 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:21,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:21,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:21,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:21,749 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:21,749 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:56:21,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:21,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:21,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:21,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:21,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:21,751 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:56:21,751 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@87c7e06[Count = 0] remaining members to acquire global barrier 2023-04-25 02:56:21,751 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:56:21,751 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:21,752 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:21,752 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:21,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:21,752 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:56:21,752 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:56:21,752 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:21,753 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:56:21,752 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:56:21,754 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:21,754 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:56:21,754 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:56:21,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:21,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:21,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:21,754 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:56:21,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:21,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:21,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:21,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:21,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:21,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:21,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:21,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:56:21,757 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:56:21,757 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:56:21,758 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:56:21,758 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:56:21,758 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:56:21,758 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:56:21,760 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:21,760 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:56:21,760 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:21,760 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:21,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:21,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:21,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:21,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:21,761 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:21,761 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:56:21,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:56:21,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:21,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:21,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:21,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:21,762 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:21,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:21,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:21,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:21,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:21,765 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:21,765 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:56:21,765 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:56:21,765 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:56:21,765 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:56:21,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:56:21,765 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:21,765 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:56:21,765 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:56:21,765 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:56:21,765 DEBUG [Thread-3656] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:56:21,765 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:21,765 DEBUG [Thread-3656] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:56:21,765 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:56:21,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:56:21,765 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:21,766 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:21,766 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:21,766 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:21,806 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230425518de32293c140dd8a625ec98dc79c2d to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425518de32293c140dd8a625ec98dc79c2d 2023-04-25 02:56:21,866 DEBUG [Thread-3656] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:56:21,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:56:21,868 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:56:21,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:56:21,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:56:21,869 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:56:21,869 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:56:21,869 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:56:21,869 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:56:21,869 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:21,869 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:21,876 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:21,876 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:21,876 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:56:21,876 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:56:21,876 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:56:21,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:56:21,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:21,877 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:21,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:56:21,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:21,877 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:56:21,878 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:56:21,878 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:56:21,878 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:56:21,878 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:56:21,878 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:56:21,878 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:56:21,878 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. 2023-04-25 02:56:21,878 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. started... 2023-04-25 02:56:21,878 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:56:21,878 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. as already flushing 2023-04-25 02:56:21,878 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. 2023-04-25 02:56:21,878 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:56:21,878 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:56:21,878 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:21,878 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:56:21,879 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:56:21,881 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:21,881 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:21,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:21,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:21,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:21,881 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:21,881 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:56:21,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:21,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:21,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:21,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:21,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:21,883 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:56:21,883 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2e3918f5[Count = 0] remaining members to acquire global barrier 2023-04-25 02:56:21,883 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:56:21,883 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:21,884 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:21,884 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:21,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:21,884 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:56:21,884 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:21,884 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:56:21,884 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:56:21,884 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:56:21,885 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:21,885 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:56:21,885 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:56:21,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:21,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:21,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:21,885 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:56:21,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:21,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:21,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:21,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:21,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:21,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:21,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:21,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:56:21,888 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:56:21,888 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:56:21,888 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:56:21,888 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:56:21,888 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:56:21,888 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:56:21,889 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:21,889 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:21,889 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:56:21,889 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:21,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:21,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:21,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:21,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:21,889 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:56:21,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:56:21,889 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:21,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:21,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:21,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:21,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:21,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:21,891 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:21,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:21,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:21,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:21,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:21,893 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:21,893 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:56:21,893 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:21,893 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:56:21,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:56:21,893 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:56:21,893 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:56:21,894 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:56:21,893 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:56:21,893 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:21,894 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:56:21,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:56:21,894 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:21,894 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:21,894 DEBUG [Thread-3656] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:56:21,894 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:56:21,894 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:21,894 DEBUG [Thread-3656] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:56:21,894 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:21,994 DEBUG [Thread-3656] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:56:21,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:56:21,996 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:56:21,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:56:21,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:56:21,998 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:56:21,998 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:56:21,998 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:56:21,998 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:56:21,998 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:21,998 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:22,000 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:56:22,000 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:22,000 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:56:22,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:56:22,000 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:22,000 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:56:22,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:22,001 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:22,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:56:22,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:22,001 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:56:22,001 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:56:22,001 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:56:22,001 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:56:22,001 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:56:22,001 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:56:22,002 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:56:22,002 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:56:22,002 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. 2023-04-25 02:56:22,002 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. started... 2023-04-25 02:56:22,002 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. as already flushing 2023-04-25 02:56:22,002 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. 2023-04-25 02:56:22,002 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:56:22,002 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:56:22,002 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:22,002 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:56:22,002 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:56:22,003 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:22,003 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:22,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:22,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:22,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:22,004 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:22,004 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:56:22,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:22,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:22,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:22,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:22,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:22,005 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:56:22,005 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@11e1f282[Count = 0] remaining members to acquire global barrier 2023-04-25 02:56:22,005 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:56:22,005 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:22,007 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:22,007 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:22,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:22,007 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:56:22,007 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:56:22,007 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:56:22,007 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:22,007 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:56:22,014 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:22,015 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:56:22,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:22,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:22,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:22,015 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:56:22,015 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:56:22,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:22,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:22,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:22,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:22,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:22,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:22,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:22,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:56:22,017 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:56:22,017 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:56:22,017 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:56:22,017 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:56:22,017 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:56:22,017 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:56:22,019 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:22,020 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:56:22,020 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:22,020 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:22,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:22,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:22,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:22,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:22,020 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:56:22,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:56:22,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:22,021 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:22,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:22,021 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:22,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39482 deadline: 1682391442019, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:22,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:22,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:22,022 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:22,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:22,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 221 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39526 deadline: 1682391442019, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:22,022 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:22,022 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:22,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:22,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39536 deadline: 1682391442020, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:22,022 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:22,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39522 deadline: 1682391442021, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:22,023 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:22,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39458 deadline: 1682391442021, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:22,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:22,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:22,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:22,025 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:22,025 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:22,025 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:22,025 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:22,025 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:56:22,025 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:22,025 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:56:22,025 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:56:22,025 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:56:22,025 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:56:22,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:56:22,025 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:22,025 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:22,025 DEBUG [Thread-3656] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:56:22,025 DEBUG [Thread-3656] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:56:22,025 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:56:22,025 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:56:22,025 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:56:22,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:56:22,125 DEBUG [Thread-3656] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:56:22,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:56:22,128 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:56:22,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:56:22,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:56:22,129 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:56:22,129 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:56:22,129 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:56:22,129 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:56:22,130 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:22,130 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:22,131 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:22,131 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:56:22,131 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:56:22,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:56:22,131 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:22,131 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:56:22,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:22,144 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:22,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:56:22,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:22,144 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:56:22,144 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:56:22,145 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:56:22,145 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:56:22,145 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:56:22,145 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:56:22,145 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:56:22,145 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. 2023-04-25 02:56:22,145 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. started... 2023-04-25 02:56:22,145 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. as already flushing 2023-04-25 02:56:22,145 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. 2023-04-25 02:56:22,145 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:56:22,145 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:56:22,145 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:22,145 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:56:22,145 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:56:22,149 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:22,149 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:22,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:22,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:22,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:22,150 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:22,150 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:56:22,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:22,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:22,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:22,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:22,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:22,151 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:56:22,151 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@45bd8020[Count = 0] remaining members to acquire global barrier 2023-04-25 02:56:22,151 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:56:22,152 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:22,153 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:22,153 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:22,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:22,153 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:56:22,153 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:56:22,153 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:22,153 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:56:22,153 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:56:22,154 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:22,154 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:56:22,154 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:56:22,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:22,154 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:56:22,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:22,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:22,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:22,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:22,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:22,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:22,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:22,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:22,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:22,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:56:22,157 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:56:22,157 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:56:22,157 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:56:22,157 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:56:22,157 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:56:22,157 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:56:22,158 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:22,158 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:22,158 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:22,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:22,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:22,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:22,158 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:56:22,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:22,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:22,159 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:22,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:22,159 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:56:22,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:56:22,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:22,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:22,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:22,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:22,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:22,161 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:22,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:22,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:22,164 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:22,164 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:22,164 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:22,164 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:22,164 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:56:22,164 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:22,164 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:56:22,164 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:56:22,164 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:56:22,164 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:56:22,164 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:56:22,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:56:22,164 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:22,164 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:22,164 DEBUG [Thread-3656] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:56:22,164 DEBUG [Thread-3656] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:56:22,165 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:56:22,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:56:22,211 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=509, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/.tmp/A/2a270a9f28c84d77914bd334c7995ad7 2023-04-25 02:56:22,221 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=509 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/.tmp/B/c05af4041b7341599ca169023f0498f1 2023-04-25 02:56:22,231 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=509 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/.tmp/C/ab1b4bba5a324ce18efc49190703d3cd 2023-04-25 02:56:22,235 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/.tmp/A/2a270a9f28c84d77914bd334c7995ad7 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/A/2a270a9f28c84d77914bd334c7995ad7 2023-04-25 02:56:22,238 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/A/2a270a9f28c84d77914bd334c7995ad7, entries=150, sequenceid=509, filesize=25.4 K 2023-04-25 02:56:22,239 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/.tmp/B/c05af4041b7341599ca169023f0498f1 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/B/c05af4041b7341599ca169023f0498f1 2023-04-25 02:56:22,242 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/B/c05af4041b7341599ca169023f0498f1, entries=150, sequenceid=509, filesize=11.9 K 2023-04-25 02:56:22,242 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/.tmp/C/ab1b4bba5a324ce18efc49190703d3cd as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/C/ab1b4bba5a324ce18efc49190703d3cd 2023-04-25 02:56:22,246 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/C/ab1b4bba5a324ce18efc49190703d3cd, entries=150, sequenceid=509, filesize=11.9 K 2023-04-25 02:56:22,247 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for d6e38fead90b2a328d380bf5577ff0ce in 861ms, sequenceid=509, compaction requested=true 2023-04-25 02:56:22,247 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for d6e38fead90b2a328d380bf5577ff0ce: 2023-04-25 02:56:22,247 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 02:56:22,247 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 02:56:22,247 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-25 02:56:22,247 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-25 02:56:22,247 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 02:56:22,248 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 79058 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-25 02:56:22,248 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HStore(1912): d6e38fead90b2a328d380bf5577ff0ce/A is initiating minor compaction (all files) 2023-04-25 02:56:22,248 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of d6e38fead90b2a328d380bf5577ff0ce/A in TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. 2023-04-25 02:56:22,248 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/A/dc713e1bf01845d2bb5c555f981ea075, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/A/40becd105c724ef18f63221c2eed5e72, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/A/2a270a9f28c84d77914bd334c7995ad7] into tmpdir=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/.tmp, totalSize=77.2 K 2023-04-25 02:56:22,249 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting dc713e1bf01845d2bb5c555f981ea075, keycount=150, bloomtype=ROW, size=26.3 K, encoding=NONE, compression=NONE, seqNum=474, earliestPutTs=1682391380420 2023-04-25 02:56:22,249 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37478 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-25 02:56:22,249 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HStore(1912): d6e38fead90b2a328d380bf5577ff0ce/B is initiating minor compaction (all files) 2023-04-25 02:56:22,249 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d6e38fead90b2a328d380bf5577ff0ce/B in TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. 2023-04-25 02:56:22,249 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/B/ed0b2eab0f3f4e6aa80995c7cfd3af38, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/B/7f453a30c1f7459ebce14d9efa60f2f3, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/B/c05af4041b7341599ca169023f0498f1] into tmpdir=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/.tmp, totalSize=36.6 K 2023-04-25 02:56:22,249 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting 40becd105c724ef18f63221c2eed5e72, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=496, earliestPutTs=1682391380738 2023-04-25 02:56:22,250 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting ed0b2eab0f3f4e6aa80995c7cfd3af38, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=474, earliestPutTs=1682391380420 2023-04-25 02:56:22,250 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] compactions.Compactor(207): Compacting 2a270a9f28c84d77914bd334c7995ad7, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=509, earliestPutTs=1682391381377 2023-04-25 02:56:22,250 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting 7f453a30c1f7459ebce14d9efa60f2f3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=496, earliestPutTs=1682391380738 2023-04-25 02:56:22,251 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting c05af4041b7341599ca169023f0498f1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=509, earliestPutTs=1682391381377 2023-04-25 02:56:22,258 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] throttle.PressureAwareThroughputController(145): d6e38fead90b2a328d380bf5577ff0ce#B#compaction#574 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 53.85 MB/second 2023-04-25 02:56:22,264 DEBUG [Thread-3656] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:56:22,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:56:22,267 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 02:56:22,269 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] throttle.PressureAwareThroughputController(145): d6e38fead90b2a328d380bf5577ff0ce#A#compaction#573 average throughput is 1.52 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-25 02:56:22,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 02:56:22,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 02:56:22,272 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 02:56:22,272 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:56:22,272 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 02:56:22,272 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 02:56:22,273 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:22,273 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:22,274 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:56:22,274 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:56:22,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:56:22,275 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:22,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:22,275 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:22,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 02:56:22,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:22,276 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 02:56:22,276 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:22,276 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 02:56:22,276 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 02:56:22,276 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 02:56:22,276 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 02:56:22,277 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 02:56:22,277 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 02:56:22,284 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 02:56:22,284 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. 2023-04-25 02:56:22,284 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 02:56:22,284 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. started... 2023-04-25 02:56:22,284 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d6e38fead90b2a328d380bf5577ff0ce 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-25 02:56:22,285 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6e38fead90b2a328d380bf5577ff0ce, store=A 2023-04-25 02:56:22,285 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:56:22,285 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6e38fead90b2a328d380bf5577ff0ce, store=B 2023-04-25 02:56:22,285 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:56:22,285 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6e38fead90b2a328d380bf5577ff0ce, store=C 2023-04-25 02:56:22,285 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:56:22,289 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/.tmp/B/06d174aac20849dbba88c5d61d29316a as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/B/06d174aac20849dbba88c5d61d29316a 2023-04-25 02:56:22,294 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in d6e38fead90b2a328d380bf5577ff0ce/B of d6e38fead90b2a328d380bf5577ff0ce into 06d174aac20849dbba88c5d61d29316a(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 02:56:22,294 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d6e38fead90b2a328d380bf5577ff0ce: 2023-04-25 02:56:22,294 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce., storeName=d6e38fead90b2a328d380bf5577ff0ce/B, priority=13, startTime=1682391382247; duration=0sec 2023-04-25 02:56:22,294 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 02:56:22,294 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-25 02:56:22,296 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37478 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-25 02:56:22,296 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HStore(1912): d6e38fead90b2a328d380bf5577ff0ce/C is initiating minor compaction (all files) 2023-04-25 02:56:22,296 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d6e38fead90b2a328d380bf5577ff0ce/C in TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. 2023-04-25 02:56:22,296 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/C/bfb26d8a29014cffada1706dc430ead0, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/C/af3e692632b44faba1a53b6f3a6331e3, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/C/ab1b4bba5a324ce18efc49190703d3cd] into tmpdir=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/.tmp, totalSize=36.6 K 2023-04-25 02:56:22,296 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting bfb26d8a29014cffada1706dc430ead0, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=474, earliestPutTs=1682391380420 2023-04-25 02:56:22,297 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting af3e692632b44faba1a53b6f3a6331e3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=496, earliestPutTs=1682391380738 2023-04-25 02:56:22,297 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] compactions.Compactor(207): Compacting ab1b4bba5a324ce18efc49190703d3cd, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=509, earliestPutTs=1682391381377 2023-04-25 02:56:22,301 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] throttle.PressureAwareThroughputController(145): d6e38fead90b2a328d380bf5577ff0ce#C#compaction#576 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-25 02:56:22,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(9158): Flush requested on d6e38fead90b2a328d380bf5577ff0ce 2023-04-25 02:56:22,526 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. as already flushing 2023-04-25 02:56:22,536 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:22,537 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:22,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39536 deadline: 1682391442532, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:22,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 218 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39458 deadline: 1682391442533, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:22,536 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:22,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39522 deadline: 1682391442532, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:22,540 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:22,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39482 deadline: 1682391442537, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:22,540 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:22,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 226 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39526 deadline: 1682391442537, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:22,639 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:22,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39536 deadline: 1682391442638, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:22,640 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:22,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39522 deadline: 1682391442638, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:22,640 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:22,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 220 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39458 deadline: 1682391442638, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:22,642 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:22,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39482 deadline: 1682391442640, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:22,643 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:22,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 228 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39526 deadline: 1682391442641, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:22,721 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/.tmp/A/66a809b9c96441babe874bca54250ee8 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/A/66a809b9c96441babe874bca54250ee8 2023-04-25 02:56:22,724 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/.tmp/C/231cfcf6b92c4aacb7646427f3657562 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/C/231cfcf6b92c4aacb7646427f3657562 2023-04-25 02:56:22,727 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in d6e38fead90b2a328d380bf5577ff0ce/A of d6e38fead90b2a328d380bf5577ff0ce into 66a809b9c96441babe874bca54250ee8(size=26.4 K), total size for store is 26.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 02:56:22,727 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for d6e38fead90b2a328d380bf5577ff0ce: 2023-04-25 02:56:22,728 INFO [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce., storeName=d6e38fead90b2a328d380bf5577ff0ce/A, priority=13, startTime=1682391382247; duration=0sec 2023-04-25 02:56:22,728 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 02:56:22,729 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in d6e38fead90b2a328d380bf5577ff0ce/C of d6e38fead90b2a328d380bf5577ff0ce into 231cfcf6b92c4aacb7646427f3657562(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 02:56:22,729 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d6e38fead90b2a328d380bf5577ff0ce: 2023-04-25 02:56:22,729 INFO [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce., storeName=d6e38fead90b2a328d380bf5577ff0ce/C, priority=13, startTime=1682391382247; duration=0sec 2023-04-25 02:56:22,729 DEBUG [RS:0;jenkins-hbase4:46697-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 02:56:22,732 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304250f61a6c9191c4eeda10ec6121203f226 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304250f61a6c9191c4eeda10ec6121203f226 2023-04-25 02:56:22,842 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:22,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39536 deadline: 1682391442841, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:22,843 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:22,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 222 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39458 deadline: 1682391442841, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:22,843 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:22,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39522 deadline: 1682391442842, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:22,845 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:22,845 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 230 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39526 deadline: 1682391442844, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:22,845 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:22,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39482 deadline: 1682391442845, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:23,141 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=529, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/.tmp/A/fc910db05e3f4953aa6a12dd8313051c 2023-04-25 02:56:23,145 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:23,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46697] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39522 deadline: 1682391443144, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:23,146 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:23,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39536 deadline: 1682391443146, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:23,152 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:23,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 224 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39458 deadline: 1682391443146, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:23,152 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:23,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 232 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39526 deadline: 1682391443146, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:23,153 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 02:56:23,153 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46697] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39482 deadline: 1682391443152, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:23,166 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=529 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/.tmp/B/939b9e55939d4934a82a9884ae2cc98c 2023-04-25 02:56:23,177 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=529 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/.tmp/C/6d26acb982e94a3d9798d33a8f21f0aa 2023-04-25 02:56:23,181 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/.tmp/A/fc910db05e3f4953aa6a12dd8313051c as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/A/fc910db05e3f4953aa6a12dd8313051c 2023-04-25 02:56:23,185 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/A/fc910db05e3f4953aa6a12dd8313051c, entries=150, sequenceid=529, filesize=25.4 K 2023-04-25 02:56:23,185 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/.tmp/B/939b9e55939d4934a82a9884ae2cc98c as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/B/939b9e55939d4934a82a9884ae2cc98c 2023-04-25 02:56:23,188 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/B/939b9e55939d4934a82a9884ae2cc98c, entries=150, sequenceid=529, filesize=11.9 K 2023-04-25 02:56:23,189 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/.tmp/C/6d26acb982e94a3d9798d33a8f21f0aa as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/C/6d26acb982e94a3d9798d33a8f21f0aa 2023-04-25 02:56:23,192 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/C/6d26acb982e94a3d9798d33a8f21f0aa, entries=150, sequenceid=529, filesize=11.9 K 2023-04-25 02:56:23,193 INFO [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=73.80 KB/75570 for d6e38fead90b2a328d380bf5577ff0ce in 909ms, sequenceid=529, compaction requested=false 2023-04-25 02:56:23,193 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d6e38fead90b2a328d380bf5577ff0ce: 2023-04-25 02:56:23,193 DEBUG [rs(jenkins-hbase4.apache.org,46697,1682391220023)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. 2023-04-25 02:56:23,193 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 02:56:23,193 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 02:56:23,193 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:23,193 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 02:56:23,193 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 02:56:23,195 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:23,195 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:23,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:23,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:23,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:23,195 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:23,195 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 02:56:23,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:23,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:23,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:23,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:23,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:23,197 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46697,1682391220023' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 02:56:23,197 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2eda5adf[Count = 0] remaining members to acquire global barrier 2023-04-25 02:56:23,197 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 02:56:23,197 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:23,202 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:23,203 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:23,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:23,203 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 02:56:23,203 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 02:56:23,203 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,46697,1682391220023' in zk 2023-04-25 02:56:23,203 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:23,203 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 02:56:23,204 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:23,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:23,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:23,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:23,205 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 02:56:23,205 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:56:23,205 DEBUG [member: 'jenkins-hbase4.apache.org,46697,1682391220023' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 02:56:23,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:23,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:23,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:23,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:23,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:23,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:23,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:23,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,46697,1682391220023': 2023-04-25 02:56:23,208 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46697,1682391220023' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 02:56:23,208 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 02:56:23,208 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 02:56:23,208 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 02:56:23,208 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 02:56:23,208 INFO [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 02:56:23,210 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:23,210 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:23,210 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:56:23,210 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:23,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:23,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:23,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 02:56:23,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 02:56:23,210 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:23,210 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:56:23,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:56:23,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:23,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 02:56:23,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:23,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 02:56:23,212 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:23,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:23,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:23,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 02:56:23,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 02:56:23,214 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:23,214 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:23,214 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 02:56:23,214 DEBUG [(jenkins-hbase4.apache.org,44265,1682391218835)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 02:56:23,214 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 02:56:23,214 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:23,214 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:23,214 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 02:56:23,214 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 02:56:23,214 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 02:56:23,215 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 02:56:23,214 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 02:56:23,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 02:56:23,214 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 02:56:23,215 DEBUG [Thread-3656] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 02:56:23,215 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 02:56:23,215 DEBUG [Thread-3656] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 02:56:23,215 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 02:56:23,215 DEBUG [Thread-3663] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x62d52321 to 127.0.0.1:62054 2023-04-25 02:56:23,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 02:56:23,215 DEBUG [Thread-3665] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2a8bf279 to 127.0.0.1:62054 2023-04-25 02:56:23,215 DEBUG [Thread-3657] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0a470a3d to 127.0.0.1:62054 2023-04-25 02:56:23,215 DEBUG [Thread-3657] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-25 02:56:23,215 DEBUG [Thread-3665] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-25 02:56:23,215 DEBUG [Thread-3663] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-25 02:56:23,217 DEBUG [Thread-3659] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4bb72437 to 127.0.0.1:62054 2023-04-25 02:56:23,217 DEBUG [Thread-3661] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3dca23c0 to 127.0.0.1:62054 2023-04-25 02:56:23,217 DEBUG [Thread-3659] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-25 02:56:23,217 DEBUG [Thread-3661] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-25 02:56:23,315 DEBUG [Thread-3656] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 02:56:23,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 02:56:23,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46697] regionserver.HRegion(9158): Flush requested on d6e38fead90b2a328d380bf5577ff0ce 2023-04-25 02:56:23,648 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing d6e38fead90b2a328d380bf5577ff0ce 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-25 02:56:23,648 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6e38fead90b2a328d380bf5577ff0ce, store=A 2023-04-25 02:56:23,648 DEBUG [Thread-3654] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2c53bf96 to 127.0.0.1:62054 2023-04-25 02:56:23,649 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:56:23,649 DEBUG [Thread-3654] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-25 02:56:23,649 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6e38fead90b2a328d380bf5577ff0ce, store=B 2023-04-25 02:56:23,649 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:56:23,649 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6e38fead90b2a328d380bf5577ff0ce, store=C 2023-04-25 02:56:23,649 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:56:23,651 DEBUG [Thread-3652] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x25baee3e to 127.0.0.1:62054 2023-04-25 02:56:23,651 DEBUG [Thread-3652] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-25 02:56:23,654 DEBUG [Thread-3646] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x062b3caa to 127.0.0.1:62054 2023-04-25 02:56:23,654 DEBUG [Thread-3646] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-25 02:56:23,655 DEBUG [Thread-3650] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x544a4215 to 127.0.0.1:62054 2023-04-25 02:56:23,655 DEBUG [Thread-3650] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-25 02:56:23,656 DEBUG [Thread-3648] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x66b5b1f3 to 127.0.0.1:62054 2023-04-25 02:56:23,656 DEBUG [Thread-3648] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-25 02:56:23,657 INFO [Listener at localhost/41915] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-25 02:56:23,657 INFO [Listener at localhost/41915] hbase.AcidGuaranteesTestTool(394): wrote 103 2023-04-25 02:56:23,657 INFO [Listener at localhost/41915] hbase.AcidGuaranteesTestTool(394): wrote 83 2023-04-25 02:56:23,657 INFO [Listener at localhost/41915] hbase.AcidGuaranteesTestTool(394): wrote 101 2023-04-25 02:56:23,657 INFO [Listener at localhost/41915] hbase.AcidGuaranteesTestTool(394): wrote 77 2023-04-25 02:56:23,657 INFO [Listener at localhost/41915] hbase.AcidGuaranteesTestTool(394): wrote 73 2023-04-25 02:56:23,657 INFO [Listener at localhost/41915] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-25 02:56:23,657 INFO [Listener at localhost/41915] hbase.AcidGuaranteesTestTool(398): read 4475 2023-04-25 02:56:23,657 INFO [Listener at localhost/41915] hbase.AcidGuaranteesTestTool(398): read 4585 2023-04-25 02:56:23,657 INFO [Listener at localhost/41915] hbase.AcidGuaranteesTestTool(398): read 4505 2023-04-25 02:56:23,657 INFO [Listener at localhost/41915] hbase.AcidGuaranteesTestTool(398): read 4491 2023-04-25 02:56:23,657 INFO [Listener at localhost/41915] hbase.AcidGuaranteesTestTool(398): read 4566 2023-04-25 02:56:23,657 INFO [Listener at localhost/41915] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-25 02:56:23,657 INFO [Listener at localhost/41915] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-25 02:56:23,657 DEBUG [Listener at localhost/41915] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x59185ba7 to 127.0.0.1:62054 2023-04-25 02:56:23,657 DEBUG [Listener at localhost/41915] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-25 02:56:23,658 INFO [Listener at localhost/41915] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-25 02:56:23,658 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-04-25 02:56:23,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure2.ProcedureExecutor(1029): Stored pid=62, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-25 02:56:23,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-25 02:56:23,661 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682391383661"}]},"ts":"1682391383661"} 2023-04-25 02:56:23,662 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-25 02:56:23,664 INFO [PEWorker-1] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-25 02:56:23,664 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=63, ppid=62, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=d6e38fead90b2a328d380bf5577ff0ce, UNASSIGN}] 2023-04-25 02:56:23,666 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=63, ppid=62, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=d6e38fead90b2a328d380bf5577ff0ce, UNASSIGN 2023-04-25 02:56:23,666 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=63 updating hbase:meta row=d6e38fead90b2a328d380bf5577ff0ce, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:23,666 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682391383666"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682391383666"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682391383666"}]},"ts":"1682391383666"} 2023-04-25 02:56:23,667 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=64, ppid=63, state=RUNNABLE; CloseRegionProcedure d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023}] 2023-04-25 02:56:23,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-25 02:56:23,819 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close d6e38fead90b2a328d380bf5577ff0ce 2023-04-25 02:56:23,819 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing d6e38fead90b2a328d380bf5577ff0ce, disabling compactions & flushes 2023-04-25 02:56:23,819 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1858): waiting for 0 compactions & cache flush to complete for region TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. 2023-04-25 02:56:23,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-25 02:56:24,060 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304258f8bf208a0844764ae3d6736f90edfec to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304258f8bf208a0844764ae3d6736f90edfec 2023-04-25 02:56:24,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-25 02:56:24,465 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=547, memsize=26.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/.tmp/A/e7f7d526fc7743a3a0f2992c23d069a4 2023-04-25 02:56:24,473 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=547 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/.tmp/B/86f0ed21b8554fa2bed979dfcfd82c59 2023-04-25 02:56:24,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-25 02:56:24,881 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=547 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/.tmp/C/4f4c65352d064c29a94e726db0761eee 2023-04-25 02:56:24,884 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/.tmp/A/e7f7d526fc7743a3a0f2992c23d069a4 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/A/e7f7d526fc7743a3a0f2992c23d069a4 2023-04-25 02:56:24,887 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/A/e7f7d526fc7743a3a0f2992c23d069a4, entries=150, sequenceid=547, filesize=25.4 K 2023-04-25 02:56:24,888 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/.tmp/B/86f0ed21b8554fa2bed979dfcfd82c59 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/B/86f0ed21b8554fa2bed979dfcfd82c59 2023-04-25 02:56:24,891 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/B/86f0ed21b8554fa2bed979dfcfd82c59, entries=150, sequenceid=547, filesize=11.9 K 2023-04-25 02:56:24,891 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/.tmp/C/4f4c65352d064c29a94e726db0761eee as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/C/4f4c65352d064c29a94e726db0761eee 2023-04-25 02:56:24,894 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/C/4f4c65352d064c29a94e726db0761eee, entries=150, sequenceid=547, filesize=11.9 K 2023-04-25 02:56:24,896 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=26.84 KB/27480 for d6e38fead90b2a328d380bf5577ff0ce in 1248ms, sequenceid=547, compaction requested=true 2023-04-25 02:56:24,896 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for d6e38fead90b2a328d380bf5577ff0ce: 2023-04-25 02:56:24,896 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. 2023-04-25 02:56:24,896 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. because compaction request was cancelled 2023-04-25 02:56:24,896 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. 2023-04-25 02:56:24,896 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 02:56:24,896 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. after waiting 0 ms 2023-04-25 02:56:24,896 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. because compaction request was cancelled 2023-04-25 02:56:24,896 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 02:56:24,896 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. 2023-04-25 02:56:24,896 DEBUG [RS:0;jenkins-hbase4:46697-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. because compaction request was cancelled 2023-04-25 02:56:24,896 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing d6e38fead90b2a328d380bf5577ff0ce 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-25 02:56:24,896 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 02:56:24,896 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6e38fead90b2a328d380bf5577ff0ce, store=A 2023-04-25 02:56:24,896 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:56:24,897 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6e38fead90b2a328d380bf5577ff0ce, store=B 2023-04-25 02:56:24,897 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:56:24,897 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6e38fead90b2a328d380bf5577ff0ce, store=C 2023-04-25 02:56:24,897 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 02:56:25,306 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230425c1c5405f25f341b58aea20866a64eba4 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425c1c5405f25f341b58aea20866a64eba4 2023-04-25 02:56:25,711 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=554, memsize=8.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/.tmp/A/2ca8f67fc725434c80fc81ce24109af0 2023-04-25 02:56:25,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-25 02:56:26,119 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=554 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/.tmp/B/21e8feceaf8f46ab95e104bbf00c9c38 2023-04-25 02:56:26,528 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=554 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/.tmp/C/46cd5561b65945b4947cffb6230a0238 2023-04-25 02:56:26,531 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/.tmp/A/2ca8f67fc725434c80fc81ce24109af0 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/A/2ca8f67fc725434c80fc81ce24109af0 2023-04-25 02:56:26,533 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/A/2ca8f67fc725434c80fc81ce24109af0, entries=150, sequenceid=554, filesize=25.4 K 2023-04-25 02:56:26,534 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/.tmp/B/21e8feceaf8f46ab95e104bbf00c9c38 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/B/21e8feceaf8f46ab95e104bbf00c9c38 2023-04-25 02:56:26,537 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/B/21e8feceaf8f46ab95e104bbf00c9c38, entries=150, sequenceid=554, filesize=11.9 K 2023-04-25 02:56:26,537 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/.tmp/C/46cd5561b65945b4947cffb6230a0238 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/C/46cd5561b65945b4947cffb6230a0238 2023-04-25 02:56:26,540 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/C/46cd5561b65945b4947cffb6230a0238, entries=150, sequenceid=554, filesize=11.9 K 2023-04-25 02:56:26,540 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=0 B/0 for d6e38fead90b2a328d380bf5577ff0ce in 1644ms, sequenceid=554, compaction requested=true 2023-04-25 02:56:26,563 DEBUG [StoreCloser-TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/A/dfb20049ffd144eca1334779fe1d1e28, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/A/969bc1313d61424db21ef2e2fcf5c054, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/A/c269266e65534535b43d63130a94f865, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/A/6e508d77291c4970bbe3d5ea0c612ece, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/A/6271daa9e1ef4f9cac76ac21c645489e, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/A/117b1fbefdb546d88cdb47a4df92e8de, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/A/6d439419060d4cd4a5bb0300afe993c6, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/A/90517b102c6b4392ad8ce64ae33c519f, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/A/da5081f299cd46f2a031f22dda953627, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/A/04ed9120bfe542a19ee411e7e2d31eb8, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/A/a1680fd4f98842418622f381b1e79ac4, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/A/e53a9f39bec94421ac6530cd94694dad, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/A/0c86fb4b21e34d7eaacef96eb63885eb, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/A/7a771c486a5f45bf8de7bb7b803f88b0, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/A/591d0a4903bf46e9b3bcacc9b62bf87e, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/A/435557a72796411297823d2e5692300e, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/A/1d6084fa00f646f4b04857d7ae1bcf77, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/A/6954056042cb40a78c6c5f586eb576a7, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/A/9cab0e4d65854095a36511ae70a61873, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/A/56a6c8b8d04444d1b77062034f787a62, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/A/2c4f81230cd1496899f70b9fcfd2d61f, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/A/c76c65d5f3b34226ba011b0a078e4762, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/A/1878815b560446ac8dda696440dbddfb, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/A/a5710dc2cdec4c5d8f7a2ee03bd56e89, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/A/e99791ccc9c54fac9e4fd7bea154c5bc, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/A/5e743c072b45416abbc4c1041bfd2d27, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/A/71799a1a2011498b9677e893d90f84fb, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/A/dc713e1bf01845d2bb5c555f981ea075, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/A/40becd105c724ef18f63221c2eed5e72, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/A/2a270a9f28c84d77914bd334c7995ad7] to archive 2023-04-25 02:56:26,564 DEBUG [StoreCloser-TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-25 02:56:26,566 DEBUG [StoreCloser-TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/A/dfb20049ffd144eca1334779fe1d1e28 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/A/dfb20049ffd144eca1334779fe1d1e28 2023-04-25 02:56:26,566 DEBUG [StoreCloser-TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/A/969bc1313d61424db21ef2e2fcf5c054 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/A/969bc1313d61424db21ef2e2fcf5c054 2023-04-25 02:56:26,567 DEBUG [StoreCloser-TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/A/c269266e65534535b43d63130a94f865 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/A/c269266e65534535b43d63130a94f865 2023-04-25 02:56:26,568 DEBUG [StoreCloser-TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/A/6e508d77291c4970bbe3d5ea0c612ece to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/A/6e508d77291c4970bbe3d5ea0c612ece 2023-04-25 02:56:26,569 DEBUG [StoreCloser-TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/A/6271daa9e1ef4f9cac76ac21c645489e to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/A/6271daa9e1ef4f9cac76ac21c645489e 2023-04-25 02:56:26,570 DEBUG [StoreCloser-TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/A/117b1fbefdb546d88cdb47a4df92e8de to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/A/117b1fbefdb546d88cdb47a4df92e8de 2023-04-25 02:56:26,571 DEBUG [StoreCloser-TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/A/6d439419060d4cd4a5bb0300afe993c6 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/A/6d439419060d4cd4a5bb0300afe993c6 2023-04-25 02:56:26,571 DEBUG [StoreCloser-TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/A/90517b102c6b4392ad8ce64ae33c519f to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/A/90517b102c6b4392ad8ce64ae33c519f 2023-04-25 02:56:26,573 DEBUG [StoreCloser-TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/A/da5081f299cd46f2a031f22dda953627 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/A/da5081f299cd46f2a031f22dda953627 2023-04-25 02:56:26,574 DEBUG [StoreCloser-TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/A/04ed9120bfe542a19ee411e7e2d31eb8 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/A/04ed9120bfe542a19ee411e7e2d31eb8 2023-04-25 02:56:26,574 DEBUG [StoreCloser-TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/A/a1680fd4f98842418622f381b1e79ac4 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/A/a1680fd4f98842418622f381b1e79ac4 2023-04-25 02:56:26,575 DEBUG [StoreCloser-TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/A/e53a9f39bec94421ac6530cd94694dad to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/A/e53a9f39bec94421ac6530cd94694dad 2023-04-25 02:56:26,576 DEBUG [StoreCloser-TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/A/0c86fb4b21e34d7eaacef96eb63885eb to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/A/0c86fb4b21e34d7eaacef96eb63885eb 2023-04-25 02:56:26,577 DEBUG [StoreCloser-TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/A/7a771c486a5f45bf8de7bb7b803f88b0 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/A/7a771c486a5f45bf8de7bb7b803f88b0 2023-04-25 02:56:26,578 DEBUG [StoreCloser-TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/A/591d0a4903bf46e9b3bcacc9b62bf87e to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/A/591d0a4903bf46e9b3bcacc9b62bf87e 2023-04-25 02:56:26,580 DEBUG [StoreCloser-TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/A/435557a72796411297823d2e5692300e to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/A/435557a72796411297823d2e5692300e 2023-04-25 02:56:26,581 DEBUG [StoreCloser-TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/A/1d6084fa00f646f4b04857d7ae1bcf77 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/A/1d6084fa00f646f4b04857d7ae1bcf77 2023-04-25 02:56:26,582 DEBUG [StoreCloser-TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/A/6954056042cb40a78c6c5f586eb576a7 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/A/6954056042cb40a78c6c5f586eb576a7 2023-04-25 02:56:26,583 DEBUG [StoreCloser-TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/A/9cab0e4d65854095a36511ae70a61873 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/A/9cab0e4d65854095a36511ae70a61873 2023-04-25 02:56:26,584 DEBUG [StoreCloser-TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/A/56a6c8b8d04444d1b77062034f787a62 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/A/56a6c8b8d04444d1b77062034f787a62 2023-04-25 02:56:26,585 DEBUG [StoreCloser-TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/A/2c4f81230cd1496899f70b9fcfd2d61f to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/A/2c4f81230cd1496899f70b9fcfd2d61f 2023-04-25 02:56:26,586 DEBUG [StoreCloser-TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/A/c76c65d5f3b34226ba011b0a078e4762 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/A/c76c65d5f3b34226ba011b0a078e4762 2023-04-25 02:56:26,587 DEBUG [StoreCloser-TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/A/1878815b560446ac8dda696440dbddfb to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/A/1878815b560446ac8dda696440dbddfb 2023-04-25 02:56:26,587 DEBUG [StoreCloser-TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/A/a5710dc2cdec4c5d8f7a2ee03bd56e89 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/A/a5710dc2cdec4c5d8f7a2ee03bd56e89 2023-04-25 02:56:26,588 DEBUG [StoreCloser-TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/A/e99791ccc9c54fac9e4fd7bea154c5bc to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/A/e99791ccc9c54fac9e4fd7bea154c5bc 2023-04-25 02:56:26,589 DEBUG [StoreCloser-TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/A/5e743c072b45416abbc4c1041bfd2d27 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/A/5e743c072b45416abbc4c1041bfd2d27 2023-04-25 02:56:26,590 DEBUG [StoreCloser-TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/A/71799a1a2011498b9677e893d90f84fb to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/A/71799a1a2011498b9677e893d90f84fb 2023-04-25 02:56:26,591 DEBUG [StoreCloser-TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/A/dc713e1bf01845d2bb5c555f981ea075 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/A/dc713e1bf01845d2bb5c555f981ea075 2023-04-25 02:56:26,592 DEBUG [StoreCloser-TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/A/40becd105c724ef18f63221c2eed5e72 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/A/40becd105c724ef18f63221c2eed5e72 2023-04-25 02:56:26,593 DEBUG [StoreCloser-TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/A/2a270a9f28c84d77914bd334c7995ad7 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/A/2a270a9f28c84d77914bd334c7995ad7 2023-04-25 02:56:26,596 DEBUG [StoreCloser-TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/B/45c27d721ee4419aaf06aae8aa2eec83, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/B/0d44dc3aecf34e1595d7ef4216834a8f, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/B/777035c54c8f48b4b5899b5d576af640, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/B/13aa7915d1df47d6909e42a1c60402a3, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/B/a0d9a6ee1308466f8489abd8adc7ded5, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/B/86fa3099f8df418eb977b338ee9a1a0a, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/B/2ea6f369063149b38f4755ea9d2aaa4a, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/B/a15f1823d59b4c43b1e56730d7903285, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/B/a57196e0e9ca474bbbc0104d2eeee473, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/B/9acecb7ac5eb44e18e76902aff7f6b28, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/B/7cec68bee0994e788f89f2e393d4fd6d, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/B/c8e03afd6945439889117a5417535654, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/B/46bdb60b0b3e4fb090f24be4a7a6713c, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/B/82c89d03598e44eca2dee92de67c927b, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/B/9362dadfefb742f7a473b4f8e3e5c47a, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/B/18bee31ffbca4c088a357b9d7c613e53, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/B/3ce395b9822f4c0da488ea66c3782d9c, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/B/0cc319688f0c49749f96a064e514af7f, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/B/9e67dc4933b8425fa704a29c92b141bf, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/B/e5632252158d4d1987c1fc9973c5a9a8, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/B/2fe830a5f0ca46d097b6ad9d1c9e7380, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/B/a2f2990f09374bf791fa5d14b7d0289b, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/B/ac5006cce1e9465db77996c879df7734, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/B/e563b5cb48194e3bb7eacda135e39c82, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/B/afa2a8fd10ed49d583f85beca482f861, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/B/69e38add9fe743eda6f8c8fe7f4edb8e, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/B/ed0b2eab0f3f4e6aa80995c7cfd3af38, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/B/86d489ab619049beb472de16eb552058, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/B/7f453a30c1f7459ebce14d9efa60f2f3, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/B/c05af4041b7341599ca169023f0498f1] to archive 2023-04-25 02:56:26,596 DEBUG [StoreCloser-TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-25 02:56:26,598 DEBUG [StoreCloser-TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/B/45c27d721ee4419aaf06aae8aa2eec83 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/B/45c27d721ee4419aaf06aae8aa2eec83 2023-04-25 02:56:26,598 DEBUG [StoreCloser-TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/B/0d44dc3aecf34e1595d7ef4216834a8f to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/B/0d44dc3aecf34e1595d7ef4216834a8f 2023-04-25 02:56:26,599 DEBUG [StoreCloser-TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/B/777035c54c8f48b4b5899b5d576af640 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/B/777035c54c8f48b4b5899b5d576af640 2023-04-25 02:56:26,600 DEBUG [StoreCloser-TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/B/13aa7915d1df47d6909e42a1c60402a3 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/B/13aa7915d1df47d6909e42a1c60402a3 2023-04-25 02:56:26,601 DEBUG [StoreCloser-TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/B/a0d9a6ee1308466f8489abd8adc7ded5 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/B/a0d9a6ee1308466f8489abd8adc7ded5 2023-04-25 02:56:26,602 DEBUG [StoreCloser-TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/B/86fa3099f8df418eb977b338ee9a1a0a to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/B/86fa3099f8df418eb977b338ee9a1a0a 2023-04-25 02:56:26,602 DEBUG [StoreCloser-TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/B/2ea6f369063149b38f4755ea9d2aaa4a to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/B/2ea6f369063149b38f4755ea9d2aaa4a 2023-04-25 02:56:26,603 DEBUG [StoreCloser-TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/B/a15f1823d59b4c43b1e56730d7903285 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/B/a15f1823d59b4c43b1e56730d7903285 2023-04-25 02:56:26,604 DEBUG [StoreCloser-TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/B/a57196e0e9ca474bbbc0104d2eeee473 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/B/a57196e0e9ca474bbbc0104d2eeee473 2023-04-25 02:56:26,605 DEBUG [StoreCloser-TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/B/9acecb7ac5eb44e18e76902aff7f6b28 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/B/9acecb7ac5eb44e18e76902aff7f6b28 2023-04-25 02:56:26,606 DEBUG [StoreCloser-TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/B/7cec68bee0994e788f89f2e393d4fd6d to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/B/7cec68bee0994e788f89f2e393d4fd6d 2023-04-25 02:56:26,607 DEBUG [StoreCloser-TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/B/c8e03afd6945439889117a5417535654 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/B/c8e03afd6945439889117a5417535654 2023-04-25 02:56:26,607 DEBUG [StoreCloser-TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/B/46bdb60b0b3e4fb090f24be4a7a6713c to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/B/46bdb60b0b3e4fb090f24be4a7a6713c 2023-04-25 02:56:26,608 DEBUG [StoreCloser-TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/B/82c89d03598e44eca2dee92de67c927b to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/B/82c89d03598e44eca2dee92de67c927b 2023-04-25 02:56:26,609 DEBUG [StoreCloser-TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/B/9362dadfefb742f7a473b4f8e3e5c47a to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/B/9362dadfefb742f7a473b4f8e3e5c47a 2023-04-25 02:56:26,610 DEBUG [StoreCloser-TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/B/18bee31ffbca4c088a357b9d7c613e53 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/B/18bee31ffbca4c088a357b9d7c613e53 2023-04-25 02:56:26,611 DEBUG [StoreCloser-TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/B/3ce395b9822f4c0da488ea66c3782d9c to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/B/3ce395b9822f4c0da488ea66c3782d9c 2023-04-25 02:56:26,612 DEBUG [StoreCloser-TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/B/0cc319688f0c49749f96a064e514af7f to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/B/0cc319688f0c49749f96a064e514af7f 2023-04-25 02:56:26,613 DEBUG [StoreCloser-TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/B/9e67dc4933b8425fa704a29c92b141bf to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/B/9e67dc4933b8425fa704a29c92b141bf 2023-04-25 02:56:26,613 DEBUG [StoreCloser-TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/B/e5632252158d4d1987c1fc9973c5a9a8 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/B/e5632252158d4d1987c1fc9973c5a9a8 2023-04-25 02:56:26,614 DEBUG [StoreCloser-TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/B/2fe830a5f0ca46d097b6ad9d1c9e7380 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/B/2fe830a5f0ca46d097b6ad9d1c9e7380 2023-04-25 02:56:26,615 DEBUG [StoreCloser-TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/B/a2f2990f09374bf791fa5d14b7d0289b to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/B/a2f2990f09374bf791fa5d14b7d0289b 2023-04-25 02:56:26,616 DEBUG [StoreCloser-TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/B/ac5006cce1e9465db77996c879df7734 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/B/ac5006cce1e9465db77996c879df7734 2023-04-25 02:56:26,616 DEBUG [StoreCloser-TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/B/e563b5cb48194e3bb7eacda135e39c82 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/B/e563b5cb48194e3bb7eacda135e39c82 2023-04-25 02:56:26,617 DEBUG [StoreCloser-TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/B/afa2a8fd10ed49d583f85beca482f861 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/B/afa2a8fd10ed49d583f85beca482f861 2023-04-25 02:56:26,618 DEBUG [StoreCloser-TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/B/69e38add9fe743eda6f8c8fe7f4edb8e to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/B/69e38add9fe743eda6f8c8fe7f4edb8e 2023-04-25 02:56:26,619 DEBUG [StoreCloser-TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/B/ed0b2eab0f3f4e6aa80995c7cfd3af38 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/B/ed0b2eab0f3f4e6aa80995c7cfd3af38 2023-04-25 02:56:26,620 DEBUG [StoreCloser-TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/B/86d489ab619049beb472de16eb552058 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/B/86d489ab619049beb472de16eb552058 2023-04-25 02:56:26,620 DEBUG [StoreCloser-TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/B/7f453a30c1f7459ebce14d9efa60f2f3 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/B/7f453a30c1f7459ebce14d9efa60f2f3 2023-04-25 02:56:26,621 DEBUG [StoreCloser-TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/B/c05af4041b7341599ca169023f0498f1 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/B/c05af4041b7341599ca169023f0498f1 2023-04-25 02:56:26,624 DEBUG [StoreCloser-TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/C/24dc7dbdd2dc4078bb27b4eb8de7f60f, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/C/5bec0d8e724d4b79a35ecf5d0a9c9b58, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/C/3d3f3b9b61a84f8db4adefcfef0d6cad, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/C/4b534914c5064cea8af561d5965651c6, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/C/bc39437696f2440cba5221a588d9b824, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/C/0d834cf5a8b74020a1839f1b56143199, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/C/ac631873d4384b54a0d9595795e44b24, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/C/a2b51c3bd3dd4888a104f7a28abe1b2f, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/C/869045a69f634dd6886f7f7c124e49a4, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/C/5995fd86db7f4e3ea56b0f11f787877e, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/C/65eebb4d60aa4ce0a416cb0b70f2c05f, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/C/1f8032eb88e94d8eb1bb42318f4fdd75, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/C/8a0db3ededdb4939a771361316e4b80b, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/C/adec822067b84a4c9fb3c85f376087e5, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/C/0fcbff231ef140a79b687e50fc9b047d, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/C/36b20ab738624ef28598ba7e9b4909cb, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/C/e275a85e2bf44fd19839f67df8bfb4d8, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/C/348fde98b0f1414e90d760f01955a23a, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/C/509e90ab9f7f44eabc903244a132da5d, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/C/40899778f8fc4d7da81c2205efe78c6b, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/C/29c6956aa568405a8b37a6c997cfe5fb, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/C/4c2d882cd65f4528a99e07514b30718e, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/C/fcfafbe2b3bd4ab782b28b2120bc2015, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/C/658553df4605456c87f70eeeb707f57b, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/C/fa941c0bffbe4e97984ea0410021fbbd, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/C/8f7df7fffc89443f9256d967bcda2ff1, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/C/bfb26d8a29014cffada1706dc430ead0, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/C/79fa302c7857479e8b8694a3dde2806d, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/C/af3e692632b44faba1a53b6f3a6331e3, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/C/ab1b4bba5a324ce18efc49190703d3cd] to archive 2023-04-25 02:56:26,624 DEBUG [StoreCloser-TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-25 02:56:26,626 DEBUG [StoreCloser-TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/C/24dc7dbdd2dc4078bb27b4eb8de7f60f to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/C/24dc7dbdd2dc4078bb27b4eb8de7f60f 2023-04-25 02:56:26,626 DEBUG [StoreCloser-TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/C/5bec0d8e724d4b79a35ecf5d0a9c9b58 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/C/5bec0d8e724d4b79a35ecf5d0a9c9b58 2023-04-25 02:56:26,627 DEBUG [StoreCloser-TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/C/3d3f3b9b61a84f8db4adefcfef0d6cad to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/C/3d3f3b9b61a84f8db4adefcfef0d6cad 2023-04-25 02:56:26,628 DEBUG [StoreCloser-TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/C/4b534914c5064cea8af561d5965651c6 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/C/4b534914c5064cea8af561d5965651c6 2023-04-25 02:56:26,629 DEBUG [StoreCloser-TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/C/bc39437696f2440cba5221a588d9b824 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/C/bc39437696f2440cba5221a588d9b824 2023-04-25 02:56:26,630 DEBUG [StoreCloser-TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/C/0d834cf5a8b74020a1839f1b56143199 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/C/0d834cf5a8b74020a1839f1b56143199 2023-04-25 02:56:26,631 DEBUG [StoreCloser-TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/C/ac631873d4384b54a0d9595795e44b24 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/C/ac631873d4384b54a0d9595795e44b24 2023-04-25 02:56:26,631 DEBUG [StoreCloser-TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/C/a2b51c3bd3dd4888a104f7a28abe1b2f to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/C/a2b51c3bd3dd4888a104f7a28abe1b2f 2023-04-25 02:56:26,632 DEBUG [StoreCloser-TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/C/869045a69f634dd6886f7f7c124e49a4 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/C/869045a69f634dd6886f7f7c124e49a4 2023-04-25 02:56:26,633 DEBUG [StoreCloser-TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/C/5995fd86db7f4e3ea56b0f11f787877e to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/C/5995fd86db7f4e3ea56b0f11f787877e 2023-04-25 02:56:26,634 DEBUG [StoreCloser-TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/C/65eebb4d60aa4ce0a416cb0b70f2c05f to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/C/65eebb4d60aa4ce0a416cb0b70f2c05f 2023-04-25 02:56:26,635 DEBUG [StoreCloser-TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/C/1f8032eb88e94d8eb1bb42318f4fdd75 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/C/1f8032eb88e94d8eb1bb42318f4fdd75 2023-04-25 02:56:26,636 DEBUG [StoreCloser-TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/C/8a0db3ededdb4939a771361316e4b80b to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/C/8a0db3ededdb4939a771361316e4b80b 2023-04-25 02:56:26,636 DEBUG [StoreCloser-TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/C/adec822067b84a4c9fb3c85f376087e5 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/C/adec822067b84a4c9fb3c85f376087e5 2023-04-25 02:56:26,637 DEBUG [StoreCloser-TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/C/0fcbff231ef140a79b687e50fc9b047d to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/C/0fcbff231ef140a79b687e50fc9b047d 2023-04-25 02:56:26,638 DEBUG [StoreCloser-TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/C/36b20ab738624ef28598ba7e9b4909cb to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/C/36b20ab738624ef28598ba7e9b4909cb 2023-04-25 02:56:26,639 DEBUG [StoreCloser-TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/C/e275a85e2bf44fd19839f67df8bfb4d8 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/C/e275a85e2bf44fd19839f67df8bfb4d8 2023-04-25 02:56:26,639 DEBUG [StoreCloser-TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/C/348fde98b0f1414e90d760f01955a23a to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/C/348fde98b0f1414e90d760f01955a23a 2023-04-25 02:56:26,640 DEBUG [StoreCloser-TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/C/509e90ab9f7f44eabc903244a132da5d to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/C/509e90ab9f7f44eabc903244a132da5d 2023-04-25 02:56:26,641 DEBUG [StoreCloser-TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/C/40899778f8fc4d7da81c2205efe78c6b to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/C/40899778f8fc4d7da81c2205efe78c6b 2023-04-25 02:56:26,642 DEBUG [StoreCloser-TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/C/29c6956aa568405a8b37a6c997cfe5fb to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/C/29c6956aa568405a8b37a6c997cfe5fb 2023-04-25 02:56:26,642 DEBUG [StoreCloser-TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/C/4c2d882cd65f4528a99e07514b30718e to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/C/4c2d882cd65f4528a99e07514b30718e 2023-04-25 02:56:26,643 DEBUG [StoreCloser-TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/C/fcfafbe2b3bd4ab782b28b2120bc2015 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/C/fcfafbe2b3bd4ab782b28b2120bc2015 2023-04-25 02:56:26,644 DEBUG [StoreCloser-TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/C/658553df4605456c87f70eeeb707f57b to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/C/658553df4605456c87f70eeeb707f57b 2023-04-25 02:56:26,645 DEBUG [StoreCloser-TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/C/fa941c0bffbe4e97984ea0410021fbbd to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/C/fa941c0bffbe4e97984ea0410021fbbd 2023-04-25 02:56:26,646 DEBUG [StoreCloser-TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/C/8f7df7fffc89443f9256d967bcda2ff1 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/C/8f7df7fffc89443f9256d967bcda2ff1 2023-04-25 02:56:26,646 DEBUG [StoreCloser-TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/C/bfb26d8a29014cffada1706dc430ead0 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/C/bfb26d8a29014cffada1706dc430ead0 2023-04-25 02:56:26,647 DEBUG [StoreCloser-TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/C/79fa302c7857479e8b8694a3dde2806d to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/C/79fa302c7857479e8b8694a3dde2806d 2023-04-25 02:56:26,648 DEBUG [StoreCloser-TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/C/af3e692632b44faba1a53b6f3a6331e3 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/C/af3e692632b44faba1a53b6f3a6331e3 2023-04-25 02:56:26,652 DEBUG [StoreCloser-TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/C/ab1b4bba5a324ce18efc49190703d3cd to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/C/ab1b4bba5a324ce18efc49190703d3cd 2023-04-25 02:56:26,655 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/recovered.edits/557.seqid, newMaxSeqId=557, maxSeqId=4 2023-04-25 02:56:26,658 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce. 2023-04-25 02:56:26,658 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for d6e38fead90b2a328d380bf5577ff0ce: 2023-04-25 02:56:26,659 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed d6e38fead90b2a328d380bf5577ff0ce 2023-04-25 02:56:26,659 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=63 updating hbase:meta row=d6e38fead90b2a328d380bf5577ff0ce, regionState=CLOSED 2023-04-25 02:56:26,659 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682391386659"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682391386659"}]},"ts":"1682391386659"} 2023-04-25 02:56:26,661 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=64, resume processing ppid=63 2023-04-25 02:56:26,661 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=64, ppid=63, state=SUCCESS; CloseRegionProcedure d6e38fead90b2a328d380bf5577ff0ce, server=jenkins-hbase4.apache.org,46697,1682391220023 in 2.9930 sec 2023-04-25 02:56:26,662 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=63, resume processing ppid=62 2023-04-25 02:56:26,662 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=63, ppid=62, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=d6e38fead90b2a328d380bf5577ff0ce, UNASSIGN in 2.9970 sec 2023-04-25 02:56:26,663 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682391386663"}]},"ts":"1682391386663"} 2023-04-25 02:56:26,664 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-25 02:56:26,665 INFO [PEWorker-1] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-25 02:56:26,666 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=62, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 3.0070 sec 2023-04-25 02:56:27,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-25 02:56:27,766 INFO [Listener at localhost/41915] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 62 completed 2023-04-25 02:56:27,767 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-04-25 02:56:27,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] procedure2.ProcedureExecutor(1029): Stored pid=65, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-25 02:56:27,769 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=65, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-25 02:56:27,770 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1227): Checking to see if procedure is done pid=65 2023-04-25 02:56:27,770 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=65, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-25 02:56:27,774 DEBUG [HFileArchiver-13] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/.tmp/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce 2023-04-25 02:56:27,776 DEBUG [HFileArchiver-13] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/.tmp/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/A, FileablePath, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/.tmp/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/B, FileablePath, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/.tmp/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/C, FileablePath, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/.tmp/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/recovered.edits] 2023-04-25 02:56:27,779 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/.tmp/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/A/2ca8f67fc725434c80fc81ce24109af0 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/A/2ca8f67fc725434c80fc81ce24109af0 2023-04-25 02:56:27,780 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/.tmp/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/A/66a809b9c96441babe874bca54250ee8 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/A/66a809b9c96441babe874bca54250ee8 2023-04-25 02:56:27,781 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/.tmp/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/A/e7f7d526fc7743a3a0f2992c23d069a4 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/A/e7f7d526fc7743a3a0f2992c23d069a4 2023-04-25 02:56:27,782 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/.tmp/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/A/fc910db05e3f4953aa6a12dd8313051c to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/A/fc910db05e3f4953aa6a12dd8313051c 2023-04-25 02:56:27,784 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/.tmp/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/B/06d174aac20849dbba88c5d61d29316a to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/B/06d174aac20849dbba88c5d61d29316a 2023-04-25 02:56:27,785 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/.tmp/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/B/21e8feceaf8f46ab95e104bbf00c9c38 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/B/21e8feceaf8f46ab95e104bbf00c9c38 2023-04-25 02:56:27,786 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/.tmp/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/B/86f0ed21b8554fa2bed979dfcfd82c59 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/B/86f0ed21b8554fa2bed979dfcfd82c59 2023-04-25 02:56:27,787 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/.tmp/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/B/939b9e55939d4934a82a9884ae2cc98c to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/B/939b9e55939d4934a82a9884ae2cc98c 2023-04-25 02:56:27,789 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/.tmp/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/C/231cfcf6b92c4aacb7646427f3657562 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/C/231cfcf6b92c4aacb7646427f3657562 2023-04-25 02:56:27,790 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/.tmp/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/C/46cd5561b65945b4947cffb6230a0238 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/C/46cd5561b65945b4947cffb6230a0238 2023-04-25 02:56:27,791 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/.tmp/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/C/4f4c65352d064c29a94e726db0761eee to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/C/4f4c65352d064c29a94e726db0761eee 2023-04-25 02:56:27,791 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/.tmp/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/C/6d26acb982e94a3d9798d33a8f21f0aa to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/C/6d26acb982e94a3d9798d33a8f21f0aa 2023-04-25 02:56:27,794 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/.tmp/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/recovered.edits/557.seqid to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce/recovered.edits/557.seqid 2023-04-25 02:56:27,795 DEBUG [HFileArchiver-13] backup.HFileArchiver(596): Deleted hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/.tmp/data/default/TestAcidGuarantees/d6e38fead90b2a328d380bf5577ff0ce 2023-04-25 02:56:27,795 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-25 02:56:27,795 DEBUG [PEWorker-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-25 02:56:27,796 DEBUG [PEWorker-5] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A] 2023-04-25 02:56:27,798 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425041d8f01945e44ca84d0202721ed0aed to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425041d8f01945e44ca84d0202721ed0aed 2023-04-25 02:56:27,799 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304250579521144a74c9c88f374a2be1fe11f to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304250579521144a74c9c88f374a2be1fe11f 2023-04-25 02:56:27,800 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042508c4c46548354735ac1f62cd70d54b41 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042508c4c46548354735ac1f62cd70d54b41 2023-04-25 02:56:27,801 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304250f61a6c9191c4eeda10ec6121203f226 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304250f61a6c9191c4eeda10ec6121203f226 2023-04-25 02:56:27,801 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304251ac3436ad08d4c2ebfa525ee015b5abd to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304251ac3436ad08d4c2ebfa525ee015b5abd 2023-04-25 02:56:27,802 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304251ba93b2f07e846a4aba7c533631c217a to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304251ba93b2f07e846a4aba7c533631c217a 2023-04-25 02:56:27,803 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304251f7f8b2b5bb549b5b722167d144e0d93 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304251f7f8b2b5bb549b5b722167d144e0d93 2023-04-25 02:56:27,804 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304252246c1fac0dd49a59575bfa6ae803c54 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304252246c1fac0dd49a59575bfa6ae803c54 2023-04-25 02:56:27,805 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042533c167e94fd8441b8294fb6cc250993f to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042533c167e94fd8441b8294fb6cc250993f 2023-04-25 02:56:27,806 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304253b9c7bdc7b054d708a558d8f84b065cf to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304253b9c7bdc7b054d708a558d8f84b065cf 2023-04-25 02:56:27,807 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304253bcbd353bbf4400e95080076864fba33 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304253bcbd353bbf4400e95080076864fba33 2023-04-25 02:56:27,808 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425514f169034b7481b8ac68acd2aa40dd8 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425514f169034b7481b8ac68acd2aa40dd8 2023-04-25 02:56:27,809 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425518de32293c140dd8a625ec98dc79c2d to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425518de32293c140dd8a625ec98dc79c2d 2023-04-25 02:56:27,810 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042566508e177ca24eeeb8cb783530f8d7aa to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042566508e177ca24eeeb8cb783530f8d7aa 2023-04-25 02:56:27,810 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425791fb55edc344a61bdd9f0f4a4085a7f to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425791fb55edc344a61bdd9f0f4a4085a7f 2023-04-25 02:56:27,811 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304258eb646c5e0cd46a5bde4dd33a3eabc36 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304258eb646c5e0cd46a5bde4dd33a3eabc36 2023-04-25 02:56:27,812 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304258f8bf208a0844764ae3d6736f90edfec to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304258f8bf208a0844764ae3d6736f90edfec 2023-04-25 02:56:27,813 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304259be9a2ecc4bf48bd91e410b991fd7634 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304259be9a2ecc4bf48bd91e410b991fd7634 2023-04-25 02:56:27,814 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425c1c5405f25f341b58aea20866a64eba4 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425c1c5405f25f341b58aea20866a64eba4 2023-04-25 02:56:27,815 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425cf1520e64ed448f2802aba7176e93c81 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425cf1520e64ed448f2802aba7176e93c81 2023-04-25 02:56:27,816 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425d1015843fc9a4d2a8a857d0cab6b2085 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425d1015843fc9a4d2a8a857d0cab6b2085 2023-04-25 02:56:27,817 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425d7a9358fbeeb4633afd89389db1b82c8 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425d7a9358fbeeb4633afd89389db1b82c8 2023-04-25 02:56:27,818 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425d8b9a92634c24331a7bfe34a6fa330b4 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425d8b9a92634c24331a7bfe34a6fa330b4 2023-04-25 02:56:27,819 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425e31609ee9b7649089fdbc27e6fbf483a to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425e31609ee9b7649089fdbc27e6fbf483a 2023-04-25 02:56:27,820 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425eda77ab9ff5d4662a22eae6a5735f123 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425eda77ab9ff5d4662a22eae6a5735f123 2023-04-25 02:56:27,821 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425f4558294ca2b41f48b498694ecc7cba2 to hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425f4558294ca2b41f48b498694ecc7cba2 2023-04-25 02:56:27,821 DEBUG [PEWorker-5] backup.HFileArchiver(596): Deleted hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-25 02:56:27,823 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=65, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-25 02:56:27,824 WARN [PEWorker-5] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-25 02:56:27,826 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-25 02:56:27,826 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=65, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-25 02:56:27,826 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-25 02:56:27,827 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1682391387827"}]},"ts":"9223372036854775807"} 2023-04-25 02:56:27,828 INFO [PEWorker-5] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-25 02:56:27,828 DEBUG [PEWorker-5] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => d6e38fead90b2a328d380bf5577ff0ce, NAME => 'TestAcidGuarantees,,1682391360128.d6e38fead90b2a328d380bf5577ff0ce.', STARTKEY => '', ENDKEY => ''}] 2023-04-25 02:56:27,828 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-25 02:56:27,828 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1682391387828"}]},"ts":"9223372036854775807"} 2023-04-25 02:56:27,829 INFO [PEWorker-5] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-25 02:56:27,831 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(130): Finished pid=65, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-25 02:56:27,831 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=65, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 64 msec 2023-04-25 02:56:27,870 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44265] master.MasterRpcServices(1227): Checking to see if procedure is done pid=65 2023-04-25 02:56:27,871 INFO [Listener at localhost/41915] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 65 completed 2023-04-25 02:56:27,882 INFO [Listener at localhost/41915] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMobGetAtomicity Thread=228 (was 229), OpenFileDescriptor=566 (was 605), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=466 (was 469), ProcessCount=170 (was 170), AvailableMemoryMB=4552 (was 4515) - AvailableMemoryMB LEAK? - 2023-04-25 02:56:27,882 INFO [Listener at localhost/41915] hbase.HBaseTestingUtility(1286): Shutting down minicluster 2023-04-25 02:56:27,882 INFO [Listener at localhost/41915] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-25 02:56:27,882 DEBUG [Listener at localhost/41915] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0f3002be to 127.0.0.1:62054 2023-04-25 02:56:27,882 DEBUG [Listener at localhost/41915] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-25 02:56:27,883 DEBUG [Listener at localhost/41915] util.JVMClusterUtil(237): Shutting down HBase Cluster 2023-04-25 02:56:27,883 DEBUG [Listener at localhost/41915] util.JVMClusterUtil(257): Found active master hash=174183398, stopped=false 2023-04-25 02:56:27,883 INFO [Listener at localhost/41915] master.ServerManager(901): Cluster shutdown requested of master=jenkins-hbase4.apache.org,44265,1682391218835 2023-04-25 02:56:27,884 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-04-25 02:56:27,884 INFO [Listener at localhost/41915] procedure2.ProcedureExecutor(629): Stopping 2023-04-25 02:56:27,884 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-04-25 02:56:27,884 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-25 02:56:27,885 DEBUG [Listener at localhost/41915] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x017faac8 to 127.0.0.1:62054 2023-04-25 02:56:27,885 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-04-25 02:56:27,885 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-04-25 02:56:27,885 DEBUG [Listener at localhost/41915] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-25 02:56:27,885 INFO [Listener at localhost/41915] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,46697,1682391220023' ***** 2023-04-25 02:56:27,885 INFO [Listener at localhost/41915] regionserver.HRegionServer(2310): STOPPED: Shutdown requested 2023-04-25 02:56:27,886 INFO [RS:0;jenkins-hbase4:46697] regionserver.HeapMemoryManager(220): Stopping 2023-04-25 02:56:27,886 INFO [RS:0;jenkins-hbase4:46697] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-04-25 02:56:27,886 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-04-25 02:56:27,886 INFO [RS:0;jenkins-hbase4:46697] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-04-25 02:56:27,886 INFO [RS:0;jenkins-hbase4:46697] regionserver.HRegionServer(3304): Received CLOSE for 4c9864611c8b9dc201fb679c2744e7a2 2023-04-25 02:56:27,887 INFO [RS:0;jenkins-hbase4:46697] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:27,887 DEBUG [RS:0;jenkins-hbase4:46697] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x30bfadd0 to 127.0.0.1:62054 2023-04-25 02:56:27,887 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 4c9864611c8b9dc201fb679c2744e7a2, disabling compactions & flushes 2023-04-25 02:56:27,887 DEBUG [RS:0;jenkins-hbase4:46697] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-25 02:56:27,887 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1682391222571.4c9864611c8b9dc201fb679c2744e7a2. 2023-04-25 02:56:27,888 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1682391222571.4c9864611c8b9dc201fb679c2744e7a2. 2023-04-25 02:56:27,888 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1682391222571.4c9864611c8b9dc201fb679c2744e7a2. after waiting 0 ms 2023-04-25 02:56:27,888 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1682391222571.4c9864611c8b9dc201fb679c2744e7a2. 2023-04-25 02:56:27,888 INFO [RS:0;jenkins-hbase4:46697] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-04-25 02:56:27,888 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 4c9864611c8b9dc201fb679c2744e7a2 1/1 column families, dataSize=78 B heapSize=488 B 2023-04-25 02:56:27,888 INFO [RS:0;jenkins-hbase4:46697] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-04-25 02:56:27,888 INFO [RS:0;jenkins-hbase4:46697] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-04-25 02:56:27,888 INFO [RS:0;jenkins-hbase4:46697] regionserver.HRegionServer(3304): Received CLOSE for 1588230740 2023-04-25 02:56:27,888 INFO [RS:0;jenkins-hbase4:46697] regionserver.HRegionServer(1475): Waiting on 2 regions to close 2023-04-25 02:56:27,889 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-04-25 02:56:27,889 DEBUG [RS:0;jenkins-hbase4:46697] regionserver.HRegionServer(1479): Online Regions={1588230740=hbase:meta,,1.1588230740, 4c9864611c8b9dc201fb679c2744e7a2=hbase:namespace,,1682391222571.4c9864611c8b9dc201fb679c2744e7a2.} 2023-04-25 02:56:27,889 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-04-25 02:56:27,889 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-04-25 02:56:27,889 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-04-25 02:56:27,889 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-04-25 02:56:27,889 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 1588230740 3/3 column families, dataSize=20.92 KB heapSize=36.24 KB 2023-04-25 02:56:27,891 DEBUG [RS:0;jenkins-hbase4:46697] regionserver.HRegionServer(1505): Waiting on 1588230740, 4c9864611c8b9dc201fb679c2744e7a2 2023-04-25 02:56:27,895 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=18.26 KB at sequenceid=93 (bloomFilter=false), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/hbase/meta/1588230740/.tmp/info/07b55a27e4e54158ad3103f5c9657bb5 2023-04-25 02:56:27,899 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 07b55a27e4e54158ad3103f5c9657bb5 2023-04-25 02:56:27,958 INFO [regionserver/jenkins-hbase4:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-04-25 02:56:28,091 DEBUG [RS:0;jenkins-hbase4:46697] regionserver.HRegionServer(1505): Waiting on 1588230740, 4c9864611c8b9dc201fb679c2744e7a2 2023-04-25 02:56:28,291 DEBUG [RS:0;jenkins-hbase4:46697] regionserver.HRegionServer(1505): Waiting on 1588230740, 4c9864611c8b9dc201fb679c2744e7a2 2023-04-25 02:56:28,295 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=78 B at sequenceid=6 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/hbase/namespace/4c9864611c8b9dc201fb679c2744e7a2/.tmp/info/bb1a84ce2d7b40fe8f9564f1b6fd332f 2023-04-25 02:56:28,300 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/hbase/namespace/4c9864611c8b9dc201fb679c2744e7a2/.tmp/info/bb1a84ce2d7b40fe8f9564f1b6fd332f as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/hbase/namespace/4c9864611c8b9dc201fb679c2744e7a2/info/bb1a84ce2d7b40fe8f9564f1b6fd332f 2023-04-25 02:56:28,303 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/hbase/namespace/4c9864611c8b9dc201fb679c2744e7a2/info/bb1a84ce2d7b40fe8f9564f1b6fd332f, entries=2, sequenceid=6, filesize=4.8 K 2023-04-25 02:56:28,304 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~78 B/78, heapSize ~472 B/472, currentSize=0 B/0 for 4c9864611c8b9dc201fb679c2744e7a2 in 416ms, sequenceid=6, compaction requested=false 2023-04-25 02:56:28,307 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=588 B at sequenceid=93 (bloomFilter=false), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/hbase/meta/1588230740/.tmp/rep_barrier/041280f0d5ab4182986804e8f4b3f777 2023-04-25 02:56:28,310 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/hbase/namespace/4c9864611c8b9dc201fb679c2744e7a2/recovered.edits/9.seqid, newMaxSeqId=9, maxSeqId=1 2023-04-25 02:56:28,310 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 041280f0d5ab4182986804e8f4b3f777 2023-04-25 02:56:28,311 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed hbase:namespace,,1682391222571.4c9864611c8b9dc201fb679c2744e7a2. 2023-04-25 02:56:28,311 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 4c9864611c8b9dc201fb679c2744e7a2: 2023-04-25 02:56:28,311 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:namespace,,1682391222571.4c9864611c8b9dc201fb679c2744e7a2. 2023-04-25 02:56:28,492 DEBUG [RS:0;jenkins-hbase4:46697] regionserver.HRegionServer(1505): Waiting on 1588230740 2023-04-25 02:56:28,663 INFO [regionserver/jenkins-hbase4:0.Chore.1] hbase.ScheduledChore(146): Chore: CompactionChecker was stopped 2023-04-25 02:56:28,663 INFO [regionserver/jenkins-hbase4:0.Chore.1] hbase.ScheduledChore(146): Chore: MemstoreFlusherChore was stopped 2023-04-25 02:56:28,692 DEBUG [RS:0;jenkins-hbase4:46697] regionserver.HRegionServer(1505): Waiting on 1588230740 2023-04-25 02:56:28,716 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=2.08 KB at sequenceid=93 (bloomFilter=false), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/hbase/meta/1588230740/.tmp/table/9901f15983e84294a95733df2e877109 2023-04-25 02:56:28,719 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 9901f15983e84294a95733df2e877109 2023-04-25 02:56:28,720 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/hbase/meta/1588230740/.tmp/info/07b55a27e4e54158ad3103f5c9657bb5 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/hbase/meta/1588230740/info/07b55a27e4e54158ad3103f5c9657bb5 2023-04-25 02:56:28,723 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 07b55a27e4e54158ad3103f5c9657bb5 2023-04-25 02:56:28,723 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/hbase/meta/1588230740/info/07b55a27e4e54158ad3103f5c9657bb5, entries=22, sequenceid=93, filesize=7.1 K 2023-04-25 02:56:28,724 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/hbase/meta/1588230740/.tmp/rep_barrier/041280f0d5ab4182986804e8f4b3f777 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/hbase/meta/1588230740/rep_barrier/041280f0d5ab4182986804e8f4b3f777 2023-04-25 02:56:28,727 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 041280f0d5ab4182986804e8f4b3f777 2023-04-25 02:56:28,727 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/hbase/meta/1588230740/rep_barrier/041280f0d5ab4182986804e8f4b3f777, entries=6, sequenceid=93, filesize=5.5 K 2023-04-25 02:56:28,727 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/hbase/meta/1588230740/.tmp/table/9901f15983e84294a95733df2e877109 as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/hbase/meta/1588230740/table/9901f15983e84294a95733df2e877109 2023-04-25 02:56:28,730 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 9901f15983e84294a95733df2e877109 2023-04-25 02:56:28,730 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/hbase/meta/1588230740/table/9901f15983e84294a95733df2e877109, entries=9, sequenceid=93, filesize=5.5 K 2023-04-25 02:56:28,730 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~20.92 KB/21417, heapSize ~36.20 KB/37064, currentSize=0 B/0 for 1588230740 in 841ms, sequenceid=93, compaction requested=false 2023-04-25 02:56:28,737 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/data/hbase/meta/1588230740/recovered.edits/96.seqid, newMaxSeqId=96, maxSeqId=1 2023-04-25 02:56:28,738 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-04-25 02:56:28,739 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-04-25 02:56:28,739 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-04-25 02:56:28,739 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:meta,,1.1588230740 2023-04-25 02:56:28,892 INFO [RS:0;jenkins-hbase4:46697] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,46697,1682391220023; all regions closed. 2023-04-25 02:56:28,900 DEBUG [RS:0;jenkins-hbase4:46697] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/oldWALs 2023-04-25 02:56:28,900 INFO [RS:0;jenkins-hbase4:46697] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C46697%2C1682391220023.meta:.meta(num 1682391222305) 2023-04-25 02:56:28,904 DEBUG [RS:0;jenkins-hbase4:46697] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/oldWALs 2023-04-25 02:56:28,905 INFO [RS:0;jenkins-hbase4:46697] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C46697%2C1682391220023:(num 1682391221821) 2023-04-25 02:56:28,905 DEBUG [RS:0;jenkins-hbase4:46697] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-25 02:56:28,905 INFO [RS:0;jenkins-hbase4:46697] regionserver.LeaseManager(133): Closed leases 2023-04-25 02:56:28,905 INFO [RS:0;jenkins-hbase4:46697] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase4:0 had [ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS, ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS] on shutdown 2023-04-25 02:56:28,905 INFO [regionserver/jenkins-hbase4:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-04-25 02:56:28,907 INFO [RS:0;jenkins-hbase4:46697] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:46697 2023-04-25 02:56:28,911 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-04-25 02:56:28,911 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,46697,1682391220023 2023-04-25 02:56:28,912 ERROR [Listener at localhost/41915-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@16481ece rejected from java.util.concurrent.ThreadPoolExecutor@127788df[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2468] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-25 02:56:28,912 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-04-25 02:56:28,912 ERROR [Listener at localhost/41915-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@3080072f rejected from java.util.concurrent.ThreadPoolExecutor@127788df[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2468] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-25 02:56:28,912 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase4.apache.org,46697,1682391220023] 2023-04-25 02:56:28,914 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase4.apache.org,46697,1682391220023; numProcessing=1 2023-04-25 02:56:28,915 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase4.apache.org,46697,1682391220023 already deleted, retry=false 2023-04-25 02:56:28,915 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase4.apache.org,46697,1682391220023 expired; onlineServers=0 2023-04-25 02:56:28,915 INFO [RegionServerTracker-0] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,44265,1682391218835' ***** 2023-04-25 02:56:28,915 INFO [RegionServerTracker-0] regionserver.HRegionServer(2310): STOPPED: Cluster shutdown set; onlineServer=0 2023-04-25 02:56:28,916 DEBUG [M:0;jenkins-hbase4:44265] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4b5cd9dd, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-04-25 02:56:28,916 INFO [M:0;jenkins-hbase4:44265] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,44265,1682391218835 2023-04-25 02:56:28,916 INFO [M:0;jenkins-hbase4:44265] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,44265,1682391218835; all regions closed. 2023-04-25 02:56:28,916 DEBUG [M:0;jenkins-hbase4:44265] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-25 02:56:28,916 DEBUG [M:0;jenkins-hbase4:44265] cleaner.LogCleaner(198): Cancelling LogCleaner 2023-04-25 02:56:28,916 WARN [OldWALsCleaner-0] cleaner.LogCleaner(186): Interrupted while cleaning old WALs, will try to clean it next round. Exiting. 2023-04-25 02:56:28,916 DEBUG [M:0;jenkins-hbase4:44265] cleaner.HFileCleaner(317): Stopping file delete threads 2023-04-25 02:56:28,916 INFO [M:0;jenkins-hbase4:44265] master.MasterMobCompactionThread(168): Waiting for Mob Compaction Thread to finish... 2023-04-25 02:56:28,916 INFO [M:0;jenkins-hbase4:44265] master.MasterMobCompactionThread(168): Waiting for Region Server Mob Compaction Thread to finish... 2023-04-25 02:56:28,916 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1682391221433] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1682391221433,5,FailOnTimeoutGroup] 2023-04-25 02:56:28,916 INFO [M:0;jenkins-hbase4:44265] hbase.ChoreService(369): Chore service for: master/jenkins-hbase4:0 had [] on shutdown 2023-04-25 02:56:28,917 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1682391221428] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1682391221428,5,FailOnTimeoutGroup] 2023-04-25 02:56:28,917 DEBUG [M:0;jenkins-hbase4:44265] master.HMaster(1512): Stopping service threads 2023-04-25 02:56:28,917 INFO [M:0;jenkins-hbase4:44265] procedure2.RemoteProcedureDispatcher(118): Stopping procedure remote dispatcher 2023-04-25 02:56:28,917 ERROR [M:0;jenkins-hbase4:44265] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] Thread[HFileArchiver-11,5,PEWorkerGroup] Thread[HFileArchiver-12,5,PEWorkerGroup] Thread[IPC Client (1612801661) connection to localhost/127.0.0.1:43711 from jenkins,5,PEWorkerGroup] Thread[HFileArchiver-13,5,PEWorkerGroup] 2023-04-25 02:56:28,918 INFO [M:0;jenkins-hbase4:44265] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-04-25 02:56:28,918 DEBUG [normalizer-worker-0] normalizer.RegionNormalizerWorker(174): interrupt detected. terminating. 2023-04-25 02:56:28,919 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-04-25 02:56:28,919 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-25 02:56:28,919 DEBUG [M:0;jenkins-hbase4:44265] zookeeper.ZKUtil(398): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2023-04-25 02:56:28,919 WARN [M:0;jenkins-hbase4:44265] master.ActiveMasterManager(326): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2023-04-25 02:56:28,919 INFO [M:0;jenkins-hbase4:44265] assignment.AssignmentManager(315): Stopping assignment manager 2023-04-25 02:56:28,919 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-04-25 02:56:28,919 INFO [M:0;jenkins-hbase4:44265] region.MasterRegion(167): Closing local region {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, isAbort=false 2023-04-25 02:56:28,920 DEBUG [M:0;jenkins-hbase4:44265] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-04-25 02:56:28,920 INFO [M:0;jenkins-hbase4:44265] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-25 02:56:28,920 DEBUG [M:0;jenkins-hbase4:44265] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-25 02:56:28,920 DEBUG [M:0;jenkins-hbase4:44265] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-04-25 02:56:28,920 DEBUG [M:0;jenkins-hbase4:44265] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-25 02:56:28,920 INFO [M:0;jenkins-hbase4:44265] regionserver.HRegion(2745): Flushing 1595e783b53d99cd5eef43b6debb2682 1/1 column families, dataSize=344.81 KB heapSize=393.51 KB 2023-04-25 02:56:28,990 DEBUG [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(152): Removing adapter for the MetricRegistry: RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-04-25 02:56:29,015 INFO [RS:0;jenkins-hbase4:46697] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,46697,1682391220023; zookeeper connection closed. 2023-04-25 02:56:29,015 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-04-25 02:56:29,015 ERROR [Listener at localhost/41915-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@48f78ced rejected from java.util.concurrent.ThreadPoolExecutor@127788df[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2468] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-25 02:56:29,016 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): regionserver:46697-0x101b93ca1070001, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-04-25 02:56:29,015 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@4d0ff476] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@4d0ff476 2023-04-25 02:56:29,016 ERROR [Listener at localhost/41915-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@7532b01 rejected from java.util.concurrent.ThreadPoolExecutor@127788df[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2468] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-25 02:56:29,016 INFO [Listener at localhost/41915] util.JVMClusterUtil(335): Shutdown of 1 master(s) and 1 regionserver(s) complete 2023-04-25 02:56:29,330 INFO [M:0;jenkins-hbase4:44265] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=344.81 KB at sequenceid=578 (bloomFilter=true), to=hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/882f3a7049b6454f812af85e0b77b93f 2023-04-25 02:56:29,334 INFO [M:0;jenkins-hbase4:44265] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 882f3a7049b6454f812af85e0b77b93f 2023-04-25 02:56:29,335 DEBUG [M:0;jenkins-hbase4:44265] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/882f3a7049b6454f812af85e0b77b93f as hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/882f3a7049b6454f812af85e0b77b93f 2023-04-25 02:56:29,338 INFO [M:0;jenkins-hbase4:44265] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 882f3a7049b6454f812af85e0b77b93f 2023-04-25 02:56:29,338 INFO [M:0;jenkins-hbase4:44265] regionserver.HStore(1080): Added hdfs://localhost:43711/user/jenkins/test-data/16f5f245-4c6e-a906-e5d5-025996fc356f/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/882f3a7049b6454f812af85e0b77b93f, entries=65, sequenceid=578, filesize=24.7 K 2023-04-25 02:56:29,338 INFO [M:0;jenkins-hbase4:44265] regionserver.HRegion(2948): Finished flush of dataSize ~344.81 KB/353082, heapSize ~393.49 KB/402936, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 418ms, sequenceid=578, compaction requested=false 2023-04-25 02:56:29,340 INFO [M:0;jenkins-hbase4:44265] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-25 02:56:29,340 DEBUG [M:0;jenkins-hbase4:44265] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-04-25 02:56:29,343 INFO [master:store-WAL-Roller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-04-25 02:56:29,343 INFO [M:0;jenkins-hbase4:44265] flush.MasterFlushTableProcedureManager(83): stop: server shutting down. 2023-04-25 02:56:29,343 INFO [M:0;jenkins-hbase4:44265] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:44265 2023-04-25 02:56:29,345 DEBUG [M:0;jenkins-hbase4:44265] zookeeper.RecoverableZooKeeper(172): Node /hbase/rs/jenkins-hbase4.apache.org,44265,1682391218835 already deleted, retry=false 2023-04-25 02:56:29,446 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-04-25 02:56:29,446 INFO [M:0;jenkins-hbase4:44265] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,44265,1682391218835; zookeeper connection closed. 2023-04-25 02:56:29,446 ERROR [Listener at localhost/41915-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@65e3c819 rejected from java.util.concurrent.ThreadPoolExecutor@8e0b885[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 4122] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-25 02:56:29,446 DEBUG [Listener at localhost/41915-EventThread] zookeeper.ZKWatcher(600): master:44265-0x101b93ca1070000, quorum=127.0.0.1:62054, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-04-25 02:56:29,446 ERROR [Listener at localhost/41915-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@6515a9a9 rejected from java.util.concurrent.ThreadPoolExecutor@8e0b885[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 4122] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-25 02:56:29,448 WARN [Listener at localhost/41915] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-04-25 02:56:29,452 INFO [Listener at localhost/41915] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-04-25 02:56:29,560 WARN [BP-829242454-172.31.14.131-1682391215512 heartbeating to localhost/127.0.0.1:43711] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-04-25 02:56:29,560 WARN [BP-829242454-172.31.14.131-1682391215512 heartbeating to localhost/127.0.0.1:43711] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-829242454-172.31.14.131-1682391215512 (Datanode Uuid 2543a0b5-98fe-4585-af6e-bed4c7220585) service to localhost/127.0.0.1:43711 2023-04-25 02:56:29,563 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/68daf6bb-f478-d813-e37b-dbe23bee4a08/cluster_35ab5f43-d241-074f-505d-0f2e954a1696/dfs/data/data1/current/BP-829242454-172.31.14.131-1682391215512] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-04-25 02:56:29,563 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/68daf6bb-f478-d813-e37b-dbe23bee4a08/cluster_35ab5f43-d241-074f-505d-0f2e954a1696/dfs/data/data2/current/BP-829242454-172.31.14.131-1682391215512] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-04-25 02:56:29,599 INFO [Listener at localhost/41915] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-04-25 02:56:29,765 INFO [Listener at localhost/41915] zookeeper.MiniZooKeeperCluster(344): Shutdown MiniZK cluster with all ZK servers 2023-04-25 02:56:29,935 INFO [Listener at localhost/41915] hbase.HBaseTestingUtility(1293): Minicluster is down